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Abstract 

This  paper  presents  a  model  for  the  general  flow  of  processing  in  the  neocortex. 

The  basic  process,  called  “sequence-seeking,”  is  a  search  for  a  sequence  of  map¬ 
pings,  or  transformations,  linking  source  and  target  representations.  The  search 
is  bi-directional,  “bottom-up”  as  well  as  “top-down,”  and  it  explores  in  parallel 
a  large  number  of  alternative  sequences.  This  operation  is  implemented  in  a 
structure  termed  “counter  streams,”  in  which  multiple  sequences  are  explored 
along  two  separate,  complementary  pathways  which  seeking  to  meet.  The  first 
part  of  the  paper  discusses  the  general  sequence-seeking  scheme  and  a  number 
of  related  processes,  such  as  the  learning  of  successful  sequences,  context  effects, 
and  the  use  of  “express  lines”  and  partial  matches.  The  second  part  discusses 
biological  implications  of  the  model  in  terms  of  connections  within  and  between 
cortical  areas.  The  model  is  compared  with  existing  data,  and  a  number  of  new 
predictions  are  proposed. 
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THE  UNIVERSAL  OPERATION  OF  THE 
CORTEX 


This  paper  presents  a  hypothesis  regarding  the  basic  scheme  of  infor¬ 
mation  processing  in  the  neocortex.  The  neocortex  (the  newer  part  of  the 
cortex)  is  a  relatively  recent  part  of  the  brain,  responsible  for  may  of  its 
advanced  fimctions.  In  humans,  it  accoimts  according  to  some  estimates 
for  as  much  as  70%  of  the  cells  in  the  central  nervous  system  (Nauta  & 
Feirtag  1986),  and  it  is  involved  in  a  variety  of  fimctions,  including  sight, 
hearing,  somatosensory  perception,  motor  planning,  language,  and  general 
aspects  of  thinking,  problem  solving,  and  planning. 

Are  there  some  general  principles  of  information  processing  that  are 
used,  with  suitable  variations,  to  carry  out  different  cortical  functions?  Our 
present  state  of  knowledge  does  not  allow  an  tmequivocal  answer  to  this  fun¬ 
damental  question,  but  the  notion  of  a  common,  basic,  cortical  scheme  has 
a  long  history,  and  has  been  discussed  recently,  e.g.,  by  Creutzfeldt  (1978), 
Barlow  (1985),  Martin  (1988),  and  White  (1989).  A  number  of  researchers 
and  Poggio  (1990)  have  proposed  different  hypotheses  at  different  levels 
of  detail  regarding  the  basic  cortical  scheme  (some  of  these  proposals  are 
reviewed  briefly  in  the  next  section). 

The  main  support  for  the  notion  of  a  “universal”  cortical  scheme  comes 
from  the  similarity  in  structure  between  different  cortical  areas  (e.g.,  visual, 
auditory,  sensory-motor,  etc.),  and  even  cortical  structures  in  different  an¬ 
imals.  As  expressed  by  Martin  (1988,  p.  639),  “...it  would  take  an  expert 
to  distinguish  rat  frontal  cortmc  flrom  sheep  parietal  cortex,  or  cat  audi¬ 
tory  cortex  from  monkey  somatosensory  cortex.”  Cell  numbers  and  the 
distribution  of  cell  types  also  appear  to  be  similar  across  species  and  in  d- 
ifferent  cortical  regions.  For  example,  Rockel  et.  ol.  (1980)  have  compared 
cell  cotmts  throughout  the  thickness  of  the  neocortex  in  different  animals, 
and  found  that,  with  the  exception  of  area  17  in  a  number  of  primates, 
cell  coimts  in  different  functional  areas  (motor,  somatic  sensory,  area  17, 
frontal,  parietal  and  temporal),  and  in  several  species  (mouse,  cat,  monkey,i 
man)  were  almost  identical. 

There  are,  of  course,  cytoarchitectonic  differences  among  cortical  areas, 
for  example,  the  striate  cortex  in  certain  primates,  or  the  giant  Betz  cells 
in  the  motor  cortex.  In  fact  a  view  expressed  by  many  neuroanatomists 
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in  the  earlier  part  of  this  century  (Broadman,  Meynert,  Voght  and  Voght, 
von  Economo,  and  others,  see,  e.g.,  in  Creutzfeldt  (1978))  suggested  that 
the  cytoarchitectonic  differences  among  cortical  areas  were  crucial  to  the 
specific  functions  of  the  different  areas. 

With  later  studies,  as  new  techniques  started  to  reveal  in  more  detail 
the  afferent  and  efferent  structures,  as  well  as  the  internal  connectivity  in 
the  cortex,  the  view  shifted  more  towards  an  emphasis  on  the  similarities 
in  structures  in  different  parts  of  the  neocortex.  A  strong  version  of  this 
view  was  expressed  by  Creutzfeldt  (1978),  who  suggested  that  “the  spe¬ 
cific  functional  and  morphological  differences  of  the  varioiis  cortical  areas 
can  be  considered  imessential.  The  ‘sp>ecific  function’  of  any  cortical  areas 
are  defined  by  the  origin  of  its  afferents  and  the  destination  of  its  efferent 
connections”  (p.  377).  Barlow  (1985)  expressed  a  similar  view:  “...re¬ 
gional  differences  of  structure  are  interesting  and  important  but  they  do 
not  violate  the  hypothesis  that  the  cortex  performs  a  uniform  operation 
throughout”  (p.  38),  and,  in  the  same  spirit,  White  (1989),  in  his  review 
of  cortical  circuits,  raises  the  notion  of  “some  common,  basic  plan”  of  the 
cerebral  cortex. 

Such  a  common,  basic  plan,  using  common  machinery,  may  be  advan¬ 
tageous  from  the  point  of  view  of  the  evolution  of  the  neocortex.  It  will 
make  it  easier  to  expand  the  capacity  of  the  cortex  by  replicating  existing 
cortical  areas  and  recruiting  the  new  areas  for  new  functions. 

It  should  be  noted,  however,  that  the  \iniformity  of  structure  and  func¬ 
tion  can  be  expected  to  hold  in  general  outline  only.  As  noted  by  Crick 
(1986):  “...it  is  reasonable  to  asstune  that  the  neocortex  has  evolved  be¬ 
cause  it  is  good  at  a  particular  sort  of  computation  that,  with  appropriate 
local  variations,  it  may  be  broadly  the  same  in  all  parts.  We  may  also  ex¬ 
pect  that  these  basic  processes  will  be  complicated  by  somewhat  elaborate 
neural  gadgetry  designed  to  make  for  better  performance”  (p.  369). 

In  addition  to  the  structural  similarities,  some  additional  support  for 
the  notion  of  a  common  cortical  scheme  comes  from  the  work  of  Sur  and  his 
collaborators  (Sur,  Garraghty  and  Roe  1988,  Pallas,  Roe,  and  Sur,  1990, 
Roe,  Hahm  and  Sur  1991)  on  re-directed  visual  input.  In  this  work,  retinal 
output  in  the  ferret  was  induced  to  project  to  the  auditory  thalamus  (MGN) 
that  projects  in  turn  to  the  primary  auditory  cortex  (Al).  An  ordered 
retinotopic  map  was  fotmd  to  be  established  in  A 1  as  a  result  of  its  novel 


3 


visual  input.  Fxirthermore,  receptive  fields  in  A 1  in  rewired  ferrets  were 
found  to  be  strikingly  similar  to  those  recorded  in  primary  visual  cortex  of 
normal  ferrets  (Roe,  1991).  These  findings  are  consistent  with  the  general 
notion  of  schemes  of  information  processing  shared  by  different  eireas  of  the 
neocortex. 

In  his  review  of  circmts  in  the  cerebral  cortex,  Martin  (1988)  suggests 
that  “..the  tempting  notion  is  then  that  nature’s  laboratory  has  hit  on  a 
process  that  enables  it  to  use  the  same  machinery  for  very  different  ends” 
(p.  640).  The  goal  of  this  paper  is  to  make  some  speculative,  but  concrete 
proposals  regarding  this  common  machinery. 

The  Cortical  Scheme:  Previous  Approaches 

Several  proposals  have  been  made  in  the  past  regarding  the  basic  oper¬ 
ation  of  the  neocortex.  In  this  section  I  briefly  list  seven  of  these  proposals. 

(i)  The  classifying  cortex.  In  Marr’s  (1970)  theory,  the  basic  function 
of  the  neocortex  is  the  classification  of  incoming  patterns.  New  classes  are 
formed  by  an  ongoing  process  of  cltistering  frequent  and  closely  similar  pat¬ 
terns.  Incoming  stimuli  are  then  tested  for  membership  in  already  existing 
classes.  The  classification  is  performed  in  two  stages.  In  the  first,  the  in¬ 
coming  pattern  is  represented  in  terms  of  so-called  “codon”  cells  that  “look 
for”  specific  sub-patterns  in  the  incoming  stimulus.  In  the  second  stage, 
“classificatory”  units,  representing  different  classes  of  events,  are  driven  by 
the  codon  cells.  The  theory  identifies  the  codon  cells  primarily  with  the 
Martinotti  cells,  a  class  of  large  layer  6  pyramidal  cells,  and  the  output  clas¬ 
sificatory  units  with  layer  3  and  5  pyramidal  cells.  Inhibitory  intemeurons 
are  assigned  two  main  tasks:  controlling  the  dynamic  range  of  the  output 
units  (by  subtractive  inhibition),  and  setting  their  threshold  (by  divisive 
inhibition). 

(ii)  Nonlinear  spatio-temporal  filter.  Creutzfeldt  (1978)  proposes  a 
view  of  the  cortical  network  as  a  nonlinear  spatio-temporal  filter  linking 
the  activity  in  the  thalamus  and  other  afferent  structures  to  the  effectors. 
This  filtering  operation  allows  through  and  emphasizes  only  certain  aspects 
of  the  signals  transmitted  to  the  cortex.  In  particular,  it  emphasizes  change, 
movement,  and  repetitive  stimulation.  In  this  scheme  the  tmiformity  of  the 
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cortex  reflects  a  uniformity  in  the  filtering  operation  applied  to  different 
afferents. 

(iii)  Coincidence  detection.  In  Barlow’s  (1985)  scheme,  the  common 
function  of  imits  throughout  the  neocortex  is  “the  detection  and  signalling 
of  suspicious  coincidence.”  A  coincidence  between  input  events  A  and  B 
is  “stispicious”  if  P(ASzB)  »  P(A)  X  P{B).  The  general  suggestion  is 
that  such  coincidences  are  used  by  the  cortex  to  form  internal  models  of 
the  environment.  Hypotheses  made  by  the  cortex  about  the  environment 
imply  certain  coincidences  among  different  inputs.  Such  coincidences  are 
in  turn  detected  and  used  to  support  or  refute  the  hypotheses.  A  simple 
Hebbian  mechanism  is  proposed  for  constructing  “coincidence  detectors”: 
synaptic  connections  to  a  given  unit  increase  in  efficacy  for  inputs  that 
co-occur  frequently. 

(iv)  Mtdti-level  relaxation.  In  Mumford’s  (1990)  theory,  multiple  cor¬ 
tical  areas,  each  representing  restricted  aspects  of  the  world,  interact  to 
achieve  a  consistent  interpretation  of  the  incoming  stimulus.  The  areas 
are  organized  hierarchically,  with  low-level  areas  concerned  with  aspects 
that  are  closer  to  the  raw  data  (e.g.,  grouping  together  line  segments),  and 
higher-level  areas  concerned  with  more  abstract  aspects  (e.g.,  identifying  a 
face).  The  interpretation  process  involves  a  relaxation  process  that  uses  two 
paths  of  information  flow:  a  cortical  one,  and  a  thalamo-cortical  one.  The 
cortical  path  involves  interactions  between  reciprocally  connected  cortical 
areas.  In  this  interaction,  the  higher,  more  abstract  area  projects  to  the 
lower  area  a  “template,”  an  instance  of  a  model  stored  in  the  higher  area 
(e.g.,  the  shape  of  a  circle  of  a  particular  size  and  location).  The  signal  from 
the  lower  to  the  higher  area  carries  a  residual:  the  remaining  differences 
between  the  data  in  the  lower  area  and  the  “hypothesis”  of  the  higher  one. 
The  relaxation  process  proceeds  xmtil  the  residual  becomes  insignificant  or 
can  be  accounted  for,  or  else  the  hypothesis  is  replaced  by  an  alternative 
one.  In  the  thalamo-cortical  loop,  subdividons  of  the  thalamus  serve  as 
so-called  “active  blackboards”  to  summarize  and  integrate  the  information 
from  the  cortical  areas.  (This  description  applies  to  the  sensory  part  of 
the  cortex;  a  related  part  of  the  theory  deals  with  actions  and  plans).  A 
somewhat  similar  proposal  is  the  “adaptive  resonance  theory”  of  Carpenter 
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and  Grossberg  (1987,  Grossberg  1988)  in  which  patterns  are  classified  by  a 
combination  bottom-up  and  top-down  processes. 

(v)  Large-scale  associative  memory.  A  number  of  models  have  pro¬ 
posed  to  view  the  cortex  primarily  as  a  large-scale  associative  memory  (or 
a  collection  of  associative  memories).  An  associative  memory  is  a  device 
in  which  a  large  number  of  patterns  (J^)  can  be  stored.  Given  an  input 
pattern,  the  system  can  retrieve,  from  all  the  stored  patterns,  that  pattern 
Pi  which  most  closely  resembles  the  input.  This  memory  has  the  power  of 
error-correction  and  completion;  the  input  pattern  may  be  a  corrupted  and 
incomplete  version  of  Pi,  and  yet  the  correct  can  be  retrieved.  Willshaw 
et  al.  (1969),  Hopfield  (1982),  and  Amit  (1989),  among  others,  have  de¬ 
scribed  possible  neural  implementations  of  associative  memories,  as  well  as 
possible  extensions  of  the  basic  memories  to  solve  a  number  of  problems. 

(vi)  Interpolating  memory.  In  Poggio’s  (1990)  HyperBF  scheme,  the 
brain  begins  the  learning  of  a  new  task  by  acquiring  a  sufficient  number  of 
input-output  examples.  For  a  new,  unknown  input,  the  appropriate  output 
is  then  derived  by  an  interpolation  (or  approximation)  process  applied  to 
the  stored  data.  For  example,  a  number  of  motor  programs,  to  reach  dif¬ 
ferent  final  configurations,  may  be  stored  in  memory.  A  new  program,  to 
reach  a  new  goal,  can  then  be  synthesized  by  approximation  in  the  space 
of  motor  programs.  (A  related  scheme  was  proposed  by  Albus  (1971)  for 
the  cerebellum.)  The  approximation  modules  in  the  brain  are  proposed 
to  be  ‘^hyperBF  modules,”  a  generalization  of  the  Radial  Basis  Fimctions 
technique  \ised  in  approximation  theory.  In  this  technique,  an  output  value 
is  obtained  by  the  superposition  of  basis  functions  stored  in  memory  and 
centered  aroimd  previously  learned  examples.  In  a  biological  implementa¬ 
tion,  the  basis  functions  are  proposed  to  be  multi-dimensional  Gaussians, 
which  can  then  be  decomposed  into  the  product  of  low-dimensional  ones. 

(vii)  Theory  of  neuronal  group  selection.  This  theory  and  its  imple¬ 
mentation  in  large  scale  computer  simulations  have  been  developed  by  G. 
Edelman  and  his  collaborators  (e.g.,  Edelman  1978,  Reeke  et  al.  1990). 

In  this  theory  the  basic  fimctional  unit  of  the  cortex  is  a  group  of  hun¬ 
dreds  to  thousands  of  strongly  interconnected  neurons.  These  basic  groups. 
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or  ‘repertoires’,  then  form  higher  order  structures  such  as  maps  and  path¬ 
ways.  Maps  along  a  pathway  and  between  parallel  pathways  intercon¬ 
nected  by  rich  reciprocal  ‘re-entrant’  connections. 

Local  rules  of  synaptic  modification  can  constantly  modify  the  connec¬ 
tions  between  groups,  as  well  as  the  structmre  of  the  imderlying  groups.  The 
process  driving  these  changes  is  likened  to  an  evolutionary  selection  pro¬ 
cess.  Groups  and  connections  that  lead  to  useful  behavioral  consequences 
will  be  favored,  leading  over  time  to  more  adaptive  behavior. 

The  theory  has  been  applied  in  simulations  to  various  problems,  includ¬ 
ing  the  organization  of  maps  in  somatosensory  cortex,  the  integration  of 
multiple  visual  maps,  and  aspects  of  pattern  recognition  and  visual-motor 
integration. 

The  current  model:  Sequence-seeking  in  counter  streams  structure  Briefiy, 
the  basic  process  in  the  model,  called  “sequence-seeking”,  is  a  search  for  a 
sequence  of  mappings,  or  transformations,  linking  source  and  target  repre¬ 
sentations.  The  search  is  bi-directional,  “bottom-up”  as  well  as“top-down”, 
exploring  simultaneously  a  large  number  of  alternatives.  This  operation  is 
implemented  in  a  structure  termed  “counter  streams”,  in  which  multiple 
sequences  are  explored  along  two  sepeu’ate,  complementary  pathways,  seek¬ 
ing  to  meet.  Roughly  speaking,  top-down  information,  originating  from 
stored  models,  fiows  along  a  descending  path,  and  bottom-up  information, 
originating  from  the  sensory  data  (in  the  case  of  perception)  flows  up  along 
an  ascending  stream.  When  similar  patterns  are  reached  from  both  ends, 
an  entire  link  connecting  the  source  and  target  representations  is  formed, 
and  also  learned  for  futme  use. 

As  we  shall  see,  the  model  proposed  in  this  paper  is  not  incompatible 
with  some  of  the  previous  proposals.  In  the  cmrent  model,  the  uniform 
operation  of  the  cortex  has  to  do  primarily  with  establishing  sequences  of 
mappings,  or  transformations,  and  it  is  not  directly  concerned  with  the 
individual  mappings  themselves.  Much  of  the  processing  is  imdoubtedly 
accomplished  within  the  individual  mappings,  perhaps  using  some  of  the 
mechanisms  described  by  the  models  listed  above.  The  model  described  be¬ 
low  is  not  concerned,  however,  with  the  computations  performed  by  specific 
mappings  in  specific  cortical  areas,  but  with  possible  common  principles 
that  do  not  depend  upon  the  details  of  individual  mappings. 
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Main  Properties  of  the  Model 

Part  I  of  the  paper  describes  the  proposed  scheme,  the  sequence-seeking 
model,  and  the  coimter-streams  structure  that  implements  it.  As  an  intro¬ 
duction,  the  main  properties  of  this  model  are  also  summarized  briefly  in 
the  list  below.  The  list  merely  introduces  the  main  notions  and  the  order 
of  presentation,  without  explanation.  The  issues  are  then  discussed  in  de¬ 
tail  in  Part  I.  Part  II  discusses  the  biological  implications  of  the  model, 
compares  it  with  existing  data,  and  proposes  new  predictions. 

•  The  basic  process,  termed  ‘sequence-seeking’,  is  a  search  for  a  sequence 
of  mappings,  or  transformations,  linking  a  source  and  target  representa¬ 
tions. 

•  The  search  is  bi-directional,  *bottom-up’  as  well  as  ‘top-down’. 

•  The  two  searches  are  performed  along  two  separate,  complementary 
pathways,  an  ascending  one  and  a  descending  one,  seeking  to  meet  (the 
‘counter  streams’  structure).  In  the  cortex,  these  two  streams  use  the  for¬ 
ward  and  backward-type  connections,  respectively. 

•  There  are  cross  connections  of  the  ‘priming’  type  between  the  two 
streams. 

•  A  track  being  traversed  in  one  stream  leaves  behind  a  primed  trace  in 
the  complementary  stream. 

•  During  the  search,  many  alternative  sequences  are  explored  in  parallel, 
using  a  maximal  possible  number  of  co-active  patterns.  Active  patterns 
persist  for  a  while,  then  they  decay  and  are  replaced  by  new  ones. 

•  When  two  sub-sequences,  one  from  ‘above’,  the  other  from  ‘below’, 
meet,  they  generate  a  chain  of  activity  linking  the  sotirce  and  target  repre¬ 
sentations. 

•  The  ascending  and  descending  streams  can  meet  at  any  level. 

•  The  entire  sequence  that  has  been  used  successfully  is  increased  in 
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relative  strength.  This  is  obtained  by  a  local  learning  rule,  based  on  the 
approximate  temporal  coincidence  of  au:tivation  in  the  two  streams. 

•  Context  effects  are  mediated  by  partial  tracks  being  primed,  for  longer 
time  intervals. 

•  ‘Express  lines'  can  traverse  more  than  a  single  level,  to  promote  the 
expansion  of  favored  sequences. 

•  FhiVorable  sequences,  determined  by  the  degree  of  match  between  the 
ascending  and  descending  streams,  will  be  expanded  further  preferentially, 
at  the  expense  of  less  promising  sequences. 

•  A  partially  successful  match  between  patterns  in  the  ascending  and 
descending  streams  is  used  to  constrain  further  transformations  of  this  pat¬ 
tern. 

•  Connections  of  different  types  (sigmiying  different  types  of  relations 
between  patterns)  are  implemented  (a)  by  using  different  sub-structures  in 
the  network,  and  (b)  by  using  associations  between  subpattems. 

•  An  additional  loop  serves  to  control  the  process,  e.g.  by  selecting 
restricted  regions  in  the  visual  field,  and  by  controlling  the  use  of  different 
sub-structures  participating  in  the  computation.  In  the  cortex,  this  loop 
may  involve  layer  5  and  subcortical  structures. 

•  Most  of  the  processing  is  performed  by  the  expanding  sequences,  rather 
than  by  a  prolonged  relaxation  or  iterative  computation  within  a  single  area. 

PART  I 

The  Sequence- Seeking  Scheme  and  Its  Imple¬ 
mentation  in  the  Counter- Streams  Structure 

The  goal  of  this  part  is  to  describe  the  proposed  scheme  of  basic  cortical 
operation,  the  sequence-seeking  scheme,  and  the  counter-streams  structure 
that  implements  it.  The  discussion  in  this  part  is  in  computational,  or 
information-processing  terms;  the  next  part  discusses  in  more  detail  the 
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proposed  biological  implementation.  The  sequence-seeking  scheme  is  pre¬ 
sented  primarily  in  the  context  of  perception,  but  the  proposal  is  intended 
to  apply  in  general  terms  to  other  cortical  functions  as  well. 

Applying  sequences  of  transformations 

In  perception,  the  brain  is  faced  with  the  task  of  establishing  a  link  be¬ 
tween  an  incoming  pattern  and  stored  representations.  The  two  will  often 
fail  to  match  exactly;  for  example,  one  may  see  a  shape  that  differs  in  posi¬ 
tion  and  size  from  previously  stored  representations  of  the  same  shape.  The 
differences  can  be  reconciled  by  applying  a  series  of  trzinsformations  to  the 
shape,  such  as  scale  and  position  changes.  A  fundamental  aspect  of  the  ba¬ 
sic  cortical  operation,  according  to  the  sequence-seeking  scheme  described 
below,  is  that  it  seeks  to  establish  a  link  between  different  representations 
or  patterns,  by  exploring  sequences  of  transformations. 

The  search  for  a  connecting  sequence  of  transformations,  mappings,  or 
states,  linking  source  and  target  representations,  coxild  provide  a  useful 
mechanism  for  various  aspects  of  perception,  as  well  as  for  other,  non- 
perceptual  functions.  For  example,  the  planning  of  a  motor  action  can 
be  cast  at  some  level  in  terms  of  seeking  a  sequence  of  possible  moves, 
linking  an  initial  configuration  with  a  desired  final  state.  Visual  routines 
(Ullman  1984),  which  are  ‘programs’  for  extracting  shape  properties  and 
spatial  relations  (such  as  “A  is  inside  B”)  have  an  analogous  structure. 
Similarly,  more  general  planning  and  problem  solving  can  often  also  be 
described  in  terms  of  establishing  a  sequence  of  transformations,  mappings, 
or  intermediate  states,  linking  some  source  and  target  representations. 

Bi-directional  search 

The  sequence-seeking  is  bi-directional,  “bottom  up”  as  well  as  “top 
down.”  In  appl}ring  a  sequence  of  transformations  to  match  an  incoming 
pattern  P  with  a  stored  pattern  M ,  the  transformations  could  be  applied 
to  P,  or  to  M,  or  to  both.  In  general,  the  only  viable  alternative  is  to 
apply  the  transformations  to  both  the  incoming  and  stored  patterns.  The 
reasons  for  the  bi-directional  search  are  the  following.  First,  some  of  the 
transformations  can  only  be  reasonably  applied  to  the  stored  pattern  M. 
Suppose  that  M  represents,  for  instance,  a  pair  of  scissors.  There  are  certain 
transformations  that  scissors  can  undergo  (opening  and  closing)  that  other 
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objects  cannot.  These  transformations  can  therefore  be  applied  to  the 
model  M  but  not  to  the  image  P.  More  imiversal  transformations  such 
as  scaling  and  rotation  can  and  should  be  applied  to  P  rather  than  to  M. 
The  reason  is  that  before  we  know  what  the  object  is,  we  attempt  to  match 
it  with  a  mnnber  of  different  stored  representations  of  different  objects. 
Applying  the  transformations  to  all  of  them  will  lead  to  a  proliferation  of 
patterns  that  can  be  avoided  by  transforming  P  alone. 

In  addition,  splitting  the  search  for  a  sequence  of  transformations  into 
two  searches  from  opposite  ends  often  increases  the  efficiency  of  the  compu¬ 
tation,  and  reduces  the  overall  niunber  of  patterns  being  explored.  These 
considerations  suggest  the  use  of  two  streams  of  transformation,  an  ascend¬ 
ing  one  (starting  at  the  stored  models)  and  a  descending  one  (starting  at 
the  image),  seeking  to  meet.  As  we  shall  see,  the  suggestion  is  that  from 
a  biological  standpoint,  these  streams  correspond  to  the  “forwmd”  zmd 
“backward”  connections  between  cortical  areas. 

Exploring  multiple  alternatives 

A  large  ntimber  of  alternative  routes  may  have  to  be  explored  before 
a  link  is  successfully  established  between  a  “sotirce”  and  a  “target”  rep¬ 
resentation.  To  achieve  fast  computation,  it  will  be  necessary  to  explore 
simultaneously  a  large  mnnber  of  alternative  routes. 

For  a  mnnber  of  reasons,  it  will  not  be  possible  to  use  the  extreme 
alternative  of  exploring  all  of  the  possible  sequences  at  the  same  time.  First, 
the  number  of  simultaneovisly  co-active  patterns  will  become  prohibitively 
large.  Second,  in  the  proposed  neural  implementation  (discussed  in  Part 
II),  patterns  are  represented  by  a  distributed  activity  over  a  population 
of  nexuons.  Since  different  patterns  are  represented  by  the  same  group  of 
neinrons,  such  a  scheme  places  limitations  on  the  mnnber  of  patterns  that 
can  be  co-active  without  leading  to  confusion  (see  Appendix).  The  system 
should  therefore  be  wired  up  in  such  as  way  that  at  any  given  moment  a 
mmcimitl  ntunber  of  alternatives  will  be  explored  in  parallel,  subject  to  the 
limitations  placed  on  cue  co-activation  of  patterns.  Patterns  will  become 
active  for  a  certain  time  period,  then  they  will  decay,  and  be  replaced  by 
alternative  patterns  that  have  not  yet  been  explored. 

In  many  models  of  visual  processing,  the  input  pattern  imdergoes  a  s- 
ingle  sequence  of  processing  stages.  In  contrast,  one  of  vhe  distinguishing 
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characteristics  of  the  sequence-seeking  scheme  is  that  an  input  pattern  gives 
rise  to  multiple  sequences  of  transformations  and  mappings  that  are  being 
explored  in  parallel.  The  terms  ‘‘transformations’'  and  “mappings”  should 
be  taken  here  in  a  broad  sense;  they  may  include  geometric  transformations 
such  as  changes  in  size,  position,  and  orientation,  the  recovery  of  different 
properties  such  as  color,  motion,  texture,  and  3-D  shape,  as  well  as  explor¬ 
ing  alternative  ways  of  representing  the  pattern  in  terms  of  its  parts  and 
its  abstract  shape  properties. 

In  exploring  multiple  sequences,  it  is  also  possible  that  a  source  and 
a  target  representation  will  be  linked  by  more  than  a  single  path.  This 
may  be  in  accord  with  the  evidence  that  different  aspects  of  the  stimulus 
may  be  analyzed  by  the  visual  cortex  along  separate  (though  interacting) 
pathways  (e.g.,  Zeki  fz  Shipp  1988).  This  possibility  is  compatible  with  the 
sequence-seeking  scheme,  but  will  not  be  discussed  here  in  detail. 

The  existence  of  multiple  alternative  sequences  will  also  increase  the 
robustness  of  the  system.  For  example,  if  a  lesion  to  the  system  disrupts 
some  established  sequences,  additional  experience  could  lead  to  the  estab¬ 
lishment  of  new,  perhaps  less  direct  sequences.  This  is  consistent  with 
many  findings  regarding  the  recovery  firom  visual  deficits  caused  by  lesions 
(e.g.  to  area  V4,  Desimone  et  al.  1990,  Schiller  &  Lee  1991).  Training 
following  the  lesion  usually  results  in  considerable  recovery  in  performance 
level.  The  recovery  is  task-specific,  and,  although  performance  improves, 
there  are  often  lasting  effects  in  terms  of  increased  response  time. 

Linking  the  ascending  and  descending  streams:  the  counter-streams 
structure 

The  scheme  that  emerges  so  far  is  summarized  schematically  in  Fig.  la. 
The  basic  operation  of  the  cortex  in  this  scheme  is  to  seek  a  sequence  of 
transformations  linking  (in  the  case  of  perception)  an  incoming  pattern  (5 
in  Fig.  la)  with  a  number  of  stored  representations  (such  as  Mi,  M2  in  Fig. 
la).  More  generally,  5  may  be  a  somce  pattern  in  one  cortical  area,  and 
Ml,  M2  target  representations  in  another.  The  search  for  a  linking  sequence 
is  bi-directional,  starting  simultaneously  at  the  top  (patterns  Mi,  M2)  and 
the  bottom  (pattern  5).  The  nodes  in  this  schematic  figure  represent  pat¬ 
terns  of  activity  (and  should  not  be  thought  of  as  individual  neurons),  and 
the  arrows  indicate  how  patterns  activate  subsequent  patterns  in  the  se- 


12 


Fiaure  1  a;  The  general  sequence-seeking  operation.  A  sequence  of  transformations 
liriking  the  source  pattern  S  with  one  of  the  target  (Q,  M1 ,  M2,  R)  representations  is 
sought  The  seardi  proceeds  by  expanding  pattern  sequences  from  both  directions. 
Nodes  in  the  figure  represent  patterns  of  acti^:  the  arrows  represent  how  patterns 
activate  siiise^rTt  patterns  in  the  sequ«ioe. 


quence.  For  example,  pattern  S  can  activate  A2,  A3,  and  A3,  and  pattern 
Ml  can  activate  B2,  B3,  B4.  In  addition  (and  not  represented  in  the  figure), 
there  are  limitations  on  the  co-activation  of  patterns;  for  example,  pattem- 
s  {B2,B3,B^)  may  be  prohibited  firom  being  co-active.  In  expanding  the 
sequence  down  from  Mi,  only  a  subset  of  these  patterns  will  be  activated 
initially,  and  will  later  decay  and  be  replaced  by  other  members  of  this 
related  group  of  patterns. 

A  linking  sequence  is  successfully  established  when  the  two  searches 
meet  somewhere  in  this  network  of  interconnected  patterns.  In  Fig.  la,  a 
linking  sequence  can  be  established  between  S  and  Mi  via  A2,  B3.  How  can 
a  successful  link  be  found  by  the  system  and  used,  e.g.,  to  select  Mi  (that 
has  been  linked  to  S)  over  M2  (that  has  not)? 

The  proposed  scheme  for  linking  the  ascending  and  descending  streams 
has  two  main  components.  First,  the  ascending  and  descending  streams 
proceed  along  separate,  complementary  pathways.  Second,  when  a  track 
is  being  traversed  in  one  stream,  it  leaves  behind  a  primed  trace  in  the 
complementaiy  stream.  This  scheme  is  shown  schematically  in  Fig.  lb. 
The  figure  is  similar  to  Fig.  la,  except  that  each  node  is  now  split  into 
two  complementary  nodes.  For  example,  B2  in  Fig.  la  is  now  split  into 
B2  on  the  ascending  pathway,  and  its  complementary  pattern  B2  on  the 
descending  one. 

The  ‘‘priming”  mentioned  above  has  the  following  meaning  in  this  struc¬ 
ture.  Suppose,  for  example,  that  the  path  from  Mi  to  A2  along  the  de¬ 
scending  path  has  been  traversed.  This  will  leave  the  complementary  path 
A2  — >  Ml  along  the  ascending  path  in  a  primed  state,  ready  to  be  activated. 
If  an  ascending  sequence  starting  at  S  will  later  reach  A2,  it  will  immedi¬ 
ately  follow  along  the  primed  path  (to  Mi),  rather  than  explore  alternative 
routes. 

The  bi-directional  search  now  proceeds  as  follows.  A  munber  of  se¬ 
quences  originating  at  S  begin  to  be  activated  along  the  ascending  path¬ 
way.  At  the  same  time,  sequences  ori^ating  at  Mi  and  M2  begin  to  expand 
downwards  along  the  descending  pathway.  Not  all  of  the  possible  sequences 
are  expanded  simultaneously,  but  whenever  a  track  (sub-sequence)  is  being 
traversed,  the  complementary  track  remains  in  a  primed  state.  Suppose 
that  by  the  time  5  has  activated  A3  along  the  ascending  stream,  the  track 
Ml  —*  S3— *  A2  had  already  been  traversed  in  the  descending  stream.  Due 
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Bgure  lb.  The  counter-stream  structure.  The  figure  is  similar  to  la,  but  each 
node  is  now  split  into  two  complementary  nodes.  Unfilled  circles  and  full 
arrows  represent  the  ascending  stream,  shaded  nodes  and  dashed  arrows 
the  descending  stream.  Every  pair  of  complementary  patterns  is  mutually 
comipcted  by  connections  of  the  priming  type  (not  shown). 


to  the  primed  traces,  the  result  will  be  the  immediate  activation  of  the 
complete  sequences  S  — »  Mi  and  Mi  — »  5.  In  this  manner,  a  complete  link 
has  been  established  between  the  representations  (Mi  and  S). 

Two  properties  of  this  linking  process  are  worth  noting.  First,  a  link  be¬ 
tween  the  ascending  and  descending  streams  can  take  pl2u:e  in  this  scheme 
at  any  level.  The  establishment  of  a  successful  link  is  accomplished  by  a 
combination  of  “bottom  up”  and  “top  down”  processing,  and  the  relative 
contribution  of  the  two  streams  is  flexible  and  can  change  with  the  circum¬ 
stances.  Second,  if  the  meeting  between  the  streams  occurs  at  a  particular 
node  {A2  in  the  example  above),  the  ascending  and  descending  patterns 
need  not  arrive  at  this  node  simultaneously.  A  meeting  is  also  possible 
between  an  active  pattern  and  a  pattern  that  hsid  been  active  some  time 
before  and  decayed,  but  left  a  primed  trace  in  the  complementary  stream. 

Fig.  2  summarizes  schematically  the  basic  link  in  this  chain  of  bi¬ 
directional  processing.  As  before,  nodes  in  this  scheme  denote  patterns  (i.e., 
activated  sub-populations  of  units).  On  the  ascending  pathway,  pattern  A 
activates  pattern  B,  and  along  the  descending  path  pattern  B  activates  A. 
Corresponding  patterns  (A  and  A,  B  and  B)  are  reciprocally  connected  by 
connections  of  the  priming  type.  This  basic  link  is  embedded  in  a  richly 
interconnected  network  in  which  each  node  is  connected  not  to  a  single, 
but  to  multiple  target  patterns. 

In  siunmaiy,  the  basic  design  of  the  sequence-seeking  model  is  relatively 
straightforward,  comprising  of  two  essentially  feed-forward  complementary 
networks  going  in  opposite  directions,  with  interaction  between  them  pri¬ 
marily  (but  not  exclxisively)  in  the  form  of  enhancing  patterns  across  the 
two  streams. 

The  biological  embodiment  of  this  scheme  is  discussed  in  more  detail 
in  Part  II.  Essentially,  the  processing  takes  place  in  two  complementary 
pathways,  with  excitatory  coimections  along  the  pathways  and  “priming” 
connections  across.  The  excitatory  connections  between  patterns  are  re¬ 
ciprocal,  obeying  the  following  general  rule;  whenever  A  is  connected  to 
B,  there  is  a  back-connection  from  B  to  A,  with  cross-connections  (of  the 
priming  type)  between  A  and  A  and  between  B  and  B.  The  reciprocity  of 
the  connections  is  an  inherent  aspect  of  the  model,  and  it  is  also  a  distin¬ 
guishing  feature  of  cortical  coimectivity. 
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Figure  2.  The  basic  unit  of  the  counter-stream  structure. 
Patterns  A  and  B  are  on  the  ascending  path,  A  and  B  on  the 
descending  path.  The  thin  arrows  between  complementary 
patterns  denote  connections  of  the  priming  type.  This  unit 
is  embedded  in  a  network  of  richly  interconnected  patterns. 


Learning  sequences 

The  linking  of  the  ascending  and  descending  streams  can  be  used  to 
learn  successful  sequences.  Since  every  pattern  node  in  a  successfiil  se¬ 
quence  will  receive  both  a  direct  activation  and  a  priming  signal  from  the 
complementary  track,  the  approximate  coincidence  of  the  two  signals  can 
be  used  to  preferentially  strengthen  the  successful  sequence. 

A  simple  learning  rule  that  will  rdnforce  successful  sequences,  therefore, 
is  the  following.  If  the  direct  activation  (e.g.,  A  —*  B  in  Fig.  2)  and  the 
priming  signal  from  the  complementary  track  (e.g.,  B  —*  B  in  Fig.  2)  are 
temporally  close,  then  the  direct  connection  {A  —*  B)  is  reinforced.  This 
rule  is  local  (it  only  depends  on  the  activation  at  B),  and  yet  it  is  sufficient 
to  reinforce  preferentially  only  successful  sequences  (that  form  a  complete 
uninterrupted  link  between  the  sovirce  and  target  patterns). 

What  will  be  the  effect  of  this  reinforcement  procedure?  As  discussed 
above,  a  given  pattern  in  the  network  can  activate  a  number  of  alternative 
successors,  not  all  of  which  can  be  activated  simultaneoxisly.  Increasing 
the  effectiveness  of  a  particular  link,  such  as  A  —*  B,  will  cause  B  to  he 
expressed  preferentially  when  A  is  activated,  before  some  of  the  competing 
alternatives.  As  a  result,  out  of  the  huge  nximber  of  potential  alternative 
paths,  sequences  that  have  proven  useful  in  the  past  will  be  explored  with 
higher  priority. 

The  number  of  alternative  sequences  that  a  realistic  size  network  can 
explore  simultaneously  is  large,  but  it  is  still  only  a  fraction  of  the  total 
nmnber  of  possible  sequences  that  can  be  represented  in  the  network.  It  is 
therefore  useful  to  increase  through  learning  the  effectiveness  of  sequences 
that  have  proven  viseful  in  the  past.  Because  the  nvunber  of  sequences  that 
can  be  explored  simultaneously  is  large,  there  is  not  much  risk  of  saturating 
the  system,  and  it  will  make  sense  to  reinforce  sequences  that  have  proven 
useful  in  the  past,  and  explore  them  again  in  the  futvure  under  similar 
circumstances. 

The  entire  process  then  relies  on  the  breadth  of  the  search,  combined 
with  the  preferences  induced  by  past  experience.  Rather  than  using  sophis¬ 
ticated  procedures  to  analyze  the  differences  between  the  patterns  being 
compared,  the  process  generates  in  parallel  a  large  number  of  alternatives, 
preferring  ones  that  proved  useful  in  the  past,  and  then  selects  the  most 
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promising  ones. 

In  the  process  of  reinforcing  successful  sequences,  changes  due  to  learn¬ 
ing  are  distributed  throughout  the  system,  and  are  not  confined  to  high 
level  centers  specializing  in  learning  (Sejnowski  1986).  Recent  studies  of 
learning  certain  perceptual  skills  suggest  that  low  level  visual  areas,  includ¬ 
ing  primary  visual  cortex,  are  indeed  involved  in  the  modifications  that 
take  place  during  the  learning  process  (Kami  &;  Sagi,  1991). 

In  addition  to  the  learning  of  complete  sequences,  as  above,  the  system 
may  also  be  engaged  in  the  learning  of  the  individual  mappings,  that  is, 
the  basic  steps  comprising  the  sequences.  This  aspect  of  the  learning  is, 
however,  outside  the  scope  of  the  current  discussion,  since  the  focus  is 
not  on  the  specifics  of  individual  processes,  but  on  their  overall  common 
structure. 

The  effect  of  context 

Context  can  have  a  powerful  influence  on  the  processing  of  visual  infor¬ 
mation  (as  well  as  in  other  perceptual  and  cognitive  domains).  A  pair  of 
similar,  elongated  blobs  in  the  image  may  be  ambiguous,  but  in  the  appro¬ 
priate  context,  e.g.,  xmder  the  bed,  they  may  be  immediately  recognized  as 
a  pair  of  slippers. 

Context  effects  can  operate  in  the  framework  of  the  sequence-seeking 
scheme  by  a  prior  priming  of  some  of  the  nodes.  The  effect  will  be  similar 
to  the  mutual  priming  of  the  ascending  and  descending  streams  but  with 
longer  time  scales.  (Priming  between  the  streams  may  last  for  tens  to 
hundreds  of  msecs.,  context  effect  should  last  for  considerably  longer,  up  to 
minutes  or  hours.)  Sequences  passing  through  the  primed  nodes  will  then 
become  facilitated.  In  the  above  example,  the  location  of  the  blobs,  under 
the  bed,  will  prime  patterns  repr^enting  objects  that  are  commonly  foimd 
in  that  location,  making  slippers  a  likely  interpretation. 

The  general  notion  of  priming  internal  representations  is  a  common 
one,  but  its  effects  in  the  framework  of  the  sequence-seeking  scheme  are 
particularly  broad.  When  certain  nodes  are  activated,  e.g.,  by  noticing  and 
identifying  the  bed  in  the  image,  they  will  initiate  sequences  of  their  own, 
and  an  entire  set  of  patterns  may  end  up  in  a  primed  state.  Later  on,  a 
large  number  of  possible  sequences  passing  through  a  primed  node  will  be 
facilitated,  compared  with  the  non-primed  sequences.  It  is  not  limited  to 
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directly  increasing  or  decreasing  the  likelihood  of  a  single  match,  but  it  can 
have  indirect,  wide-spread  effects,  by  facilitating  otherwise  less  favorable 
sequences.  A  context  pattern  A  may  help  to  bring  about  the  activation 
of  B,  not  because  A  and  B  have  been  directly  associated  in  the  past,  but 
because  A  may  have  a  sequence  leading  to  some  intermediate  pattern  C, 
and,  later  on,  a  currently  activated  pattern  may  have  another  sequence 
leading  to  B  via  the  primed  pattern  C.  (Context  may  possibly  involve  also 
inhibitoiy  effects,  making  some  of  the  paths  less  likely,  but  this  will  not  be 
discussed  here  further.) 

These  general  characteristics  capture  some  of  the  fundamental  aspects 
of  context  effects  in  humans.  Humans’  perception  and  cognition  appear  to 
have  an  almost  imcanny  capacity  (which  is  extremely  difficult  to  reproduce 
in  artificial  systems)  for  bringing  in  relevant  context  information  in  a  broad 
and  fiexible  manner.  The  broad,  indirect  effects  may  be  reproduced  in  part 
by  context  effects  in  the  sequence-seeking  scheme. 

Express  lines 

In  expanding  the  descending  sequences,  how  can  the  initial  selection 
of  models  be  performed?  Clearly,  to  cut  down  the  number  of  competing 
sequences  in  the  descending  stream,  it  would  be  tiseful  to  expand  with 
higher  priority  alternatives  that  appear  more  promising.  For  example,  in 
attempting  to  recognize  an  object,  some  models  (a  face,  say)  may  become 
more  likely  than  others  on  the  basis  of  partial  analysis,  although  it  may  not 
be  possible  to  identify  the  individual.  It  would  be  advantageous  under  these 
circumstances  to  expand  many  face-related  sequences,  possibly  at  the  ex¬ 
pense  of  others.  Such  an  effect  can  be  obtained  by  “express  lines,”  directly 
connecting  low-level  to  higher-level  nodes  along  the  ascending  stream.  The 
express  lines  will  activate  (directly  or  indirectly)  patterns  on  the  descend¬ 
ing  path.  This  will  initiate  an  expansion  of  sequences  firom  the  selected 
patterns.  This  selection  of  higher-level  patterns  can  be  viewed  as  invoking 
a  hypothecs  suggested  by  the  data,  but  which  has  yet  to  be  confirmed.  A 
link  to  the  ascending  stream  will  still  be  required  to  confirm  the  hypothesis. 
Note  that,  unlike  the  priming  interaction  between  streams,  in  the  case  of 
express  lines  the  ascending  stream  can  directly  activate  the  descending  one. 
Express  lines  coiild  also  use  inhibition  rather  than  facilitation;  if  the  par¬ 
tially  expanded  sequences  in  the  ascending  stream  render  some  higher-level 
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nodes  \inlikely,  inhibitory  ‘‘express  lines”  could  be  used  to  suppress  their 
expansion  in  the  descending  stream. 

The  express  lines  provide  one  mechanism  for  “indexing”  into  the  large 
number  of  models  stored  in  memory,  that  is,  for  the  initial  selection  of 
likely  models  on  the  descending  stream.  This  initial  selection  of  models  is 
not  only  to  a  single  “topmost”  levd,  models  at  different  levels  along  the 
descending  stream  can  also  be  “indexed”  and  serve  as  the  starting  point 
for  descending  sub-sequences. 

Another  mechanism  for  model  selection  is  provided  by  the  effects  of 
expectation  and  context.  Knowledge  about  the  current  situation  can  lead 
to  the  activation  or  priming  of  a  subset  of  models  that  will  then  become 
preferential  sources  for  descending  sequences.  The  set  of  active  models  will 
then  be  modified  and  refined  throughout  the  sequence-seeking  process,  as 
described  in  the  next  section. 

Refining  the  expansion 

Favorable  sequences  cotdd  also  be  expanded  preferentially  in  another 
manner,  based  on  the  degree  of  match  between  the  ascending  and  descend¬ 
ing  streams.  For  this  process  of  refining  promising  sequences  the  linking 
between  the  streams  should  act  as  a  graded  event,  depending  on  the  de¬ 
gree  of  match  between  corresponding  patterns.  That  is,  the  assumption  is 
that  if  the  two  streams  lead  to  sufficiently  similar  patterns,  then  the  com¬ 
bined  sequence  will  be  activated,  and  the  activation  will  reflect  the  degree 
of  match  between  the  corresponding  patterns.  As  a  result,  sequences  with 
higher  degree  of  match  will  have  a  higher  probability  of  being  expanded  in 
subsequent  steps. 

This  will  lead  to  a  process  by  which  coarse  matches  are  established  first, 
followed  by  successive  refinements.  Initially,  the  space  of  possible  sequences 
will  be  sampled  coarsely,  subject  to  the  limitations  on  the  number  of  co¬ 
active  sequences.  The  two  populations  of  partial  sequences,  along  the  two 
streams,  will  then  interact,  by  mutually  comparing  patterns  and  reinforc¬ 
ing  favorable  ones.  The  more  promising  sequences  (the  ones  leading  to 
better  matches  between  the  ascending  and  descending  sequences)  will  then 
serve  as  preferred  starting  points  for  exploring  additional  sub-sequences. 
In  this  manner,  the  sequence  seeking  process  will  seek  modifications  and 
refinements  of  the  more  promising  sequences. 
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This  process  has  some  features  in  common  with  a  family  of  optimiza¬ 
tion  and  search  procedures  known  as  “genetic  algorithms”  (Holland  1975), 
that  have  been  shown  in  some  recent  evaluations  to  behave  quite  efficiently 
compared  with  alternative  methods,  including  a  number  of  relaxation  pro¬ 
cesses,  gradient  descent,  simulated  annealing,  and  heuristic  search  (Brsuly 
1985,  Peterson  1990). 

Constrained  transformations 

As  the  ascending  and  descending  streams  are  being  expanded,  they  may 
sometimes  reach  a  partial  rather  than  a  complete  match.  For  example,  in 
an  image  of  scissors,  a  complete  match  may  be  obtained  for  one  blade 
but  not  for  the  other.  More  generally,  if  an  object  has  been  distorted  in  a 
flexible,  non-rigid  manner,  a  sequence  of  transformations  may  lead  to  a  good 
match  of  a  part  of  the  object  but  not  all  of  it.  When  this  occurs,  it  would 
be  advantageotis  to  maintain  the  partially  successful  match  in  subsequent 
transformations  of  the  object.  The  unmatched  part  will  transform,  but  the 
matched  part  will  remain  “frozen.”  The  question  of  how  such  constrained 
transformations  may  be  obtained  is  an  interesting  one,  but  will  not  be 
considered  further  here.  Let  me  note,  however,  that  such  a  process  could 
play  a  useful  role  in  the  recognition  of  objects  imdergoing  complex,  non- 
rigid  distortions,  a  task  that  proved  to  be  one  of  the  most  difficult  aspects 
of  theories  of  visual  object  recognition. 

Controlling  the  sequence-seeking  process 

The  sequence-seeking  process  is  likely  to  include  control  mechanisms 
that  help  to  restrict  and  coordinate  the  sequences  being  explored.  Perhaps 
the  most  straightforward  example  of  such  a  mechanism  in  visual  perception 
is  spatial  selection,  i.e.,  restricting  the  region  in  the  incoming  image  that  is 
being  explored.  There  is  substantial  evidence  in  visual  perception  for  the 
existence  of  such  processes.  A  popular  metaphor  has  been  the  “spotlight”  of 
attention  (Posner  et  al.  1980).  Regions  inade  this  “spotlight”  are  selected 
for  more  extensive  processing.  I  wiU  not  review  here  the  evidence  and 
theories  related  to  these  selection  processes.  For  the  current  discussion,  the 
miiin  point  is  that  selection  processes  of  this  type  will  serve  to  select  and 
control  the  expanding  sequences  in  the  ascending  stream.  In  a  biological 
implementation,  discussed  in  Part  II,  it  is  suggested  that  a  special  control 
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loop,  possibly  involving  layer  5  of  the  cortex  and  subcortical  structures  may 
be  involved  in  this  control  process. 

Different  types  of  relations 

In  the  sequence-seeking  scheme,  a  sequence  is  expanded  when  a  pattern 
ctirrently  at  the  tail  end  of  the  sequence  activates  new  patterns  connected 
to  or  associated  with  it.  Connections  between  patterns  may  be  used  to 
signify  that  the  two  are  related,  but  relations  come  in  different  t}rpes.  For 
example,  the  notion  of  a  “dog”  is  associated  with  specific  kinds  of  dogs, 
but  also  with  entirely  different  notions  (such  as  a  leash).  When  attempting 
to  recognize  a  dog,  some  of  the  connections  are  relevant  and  should  be 
activated,  while  others  are  not.  This  is  a  general  problem  in  many  models; 
it  is  not  enough  to  associate  different  representations,  it  is  also  important 
to  make  at  least  some  distinctions  between  different  types  of  associations. 
What  can  distinguish  in  a  network  between  different  types  of  association? 

One  common  solution  is  to  assume  that  connections  somehow  come  in 
different  “flavors”  or  types.  There  is  one  type  specifying,  e.g.,  being-part- 
of,  another  signifying  sub-categories,  etc.  Different  types  of  connections 
can  be  readily  implemented  in  artificial  systems,  but  are  more  difficxilt  to 
envisage  in  a  biological  setting. 

I  will  not  discuss  possible  solutions  in  detail,  partly  because  I  suspect 
that  there  may  not  be  a  single  ample  solution.  Instead,  I  will  mention 
briefly  two  possible  aspects  of  the  solution.  Part  of  the  solution  may  be 
that  different  types  of  relations  are  implemented  in  different  sub-networks 
in  the  system.  For  example,  there  may  be  one  network  in  one  cortical 
area  where  the  connections  can  be  interpreted  as  specifying  sub-categories, 
whereas  “being-part-of”  or  “sounds-like,”  etc.  are  implemented  in  different 
sub-networks  and  different  cortical  areas.  One  of  the  tasks  of  the  control 
processes  mentioned  in  the  previous  section  may  then  be  to  enable  and 
disable  sub-networks  as  reqiiired. 

A  second  aspect  of  the  solution  may  come  from  the  use  of  spatial  s- 
election,  discussed  in  the  previous  section,  as  illustrated  by  the  following 
example.  It  was  mentioned  above  how  context  information  may  affect  the 
flow  of  the  expanding  sequences,  helping  to  identify,  for  instance,  a  pair 
of  elongated  blobs  under  the  bed  as  a  pair  of  slippers.  In  this  example, 
“bed”  and  “slippers”  are  associated  in  memory,  but  it  is  a  special  type 
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of  association.  The  notion  of  “bed”  is  also  associated  with  a  l2u:ge  num¬ 
ber  of  other  notions  such  as  “sofa”  and  “furniture”  which,  unlike  slippers, 
are  not  associated  with  being  xmder  the  bed.  One  can  imagine  a  useful 
scheme  in  which  a  pattern  representing  a  bed  is  associated  as  a  whole  in 
an  associative  memory  with  a  number  of  other  patterns  related  to  “bed.” 
In  addition,  a  sub-pattern  corresponding  to  the  location  under  the  bed  is 
associated,  e.g.,  with  the  notion  of  slippers.  Here  is  where  spatial  selection 
can  play  a  useful  role.  If  during  the  processing  the  entire  “bed”  pattern 
is  activated,  various  patterns  associated  with  “bed”  will  become  activated 
next.  If,  however,  the  sub-pattern  representing  the  region  imder  the  bed  is 
selected,  items  associated  with  being  under  the  bed  will  be  selected  next. 
As  this  example  illustrates,  connections  between  sub-patterns  can  be  used 
to  specify  different  types  of  connections. 

In  concluding  this  part,  it  will  be  useful  to  follow  in  general  outline  the 
flow  of  the  sequence-seeking  process,  for  example  in  the  case  of  the  visual 
identification  of  a  familiar  shape.  The  processing  and  analysis  of  the  in¬ 
put  pattern  will  center  in  this  scheme  around  the  bi-directional  search  for  a 
linking  sequence  of  mappings  between  a  source  and  a  target  representation, 
and  it  will  include  the  following  stages.  The  input  pattern  will  initiate  se¬ 
quences  of  transformations  and  mappings,  expanding  in  a  bottom-up  man¬ 
ner.  Rather  than  a  single  stream  of  processing  stages,  multiple  alternatives 
will  be  explored  simultaneously.  These  sequences  will  include  geometric 
transformations,  such  as  changes  in  position,  orientation,  and  size,  and  al¬ 
so  transformations  and  mappings  that  are  not  geometric  in  nature,  such  as 
exploring  alternative  ways  of  representing  and  abstracting  the  input  shape. 
The  expanding  sequences  will  remain  active  for  a  while,  then  decay  and  be 
replaced  by  alternative  ones.  The  sequences  being  explored  will  leave  the 
complementary  descending  tracks  in  a  primed  state. 

At  the  same  time,  sequences  will  be  expanded  in  a  top-down  manner, 
starting  from  previously  stored  patterns.  The  transformations  applied  to 
the  stored  representations  will  be  shaped  past  experience  with  these 
patterns.  The  descending  sequences  will  also  leave  their  complementary 
tracks  in  a  primed  state. 

Some  of  the  descending  sequences  will  be  favored  by  ‘express  lines’  from 
the  ascending  patterns,  and  will  be  expanded  preferentially,  based  on  some 
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properties  of  the  shape,  such  as  its  color,  overall  elongation,  curvatiire 
properties  etc.  The  expansion  will  also  be  biased  by  context  effects  (e.g. 
by  the  priming  of  sub-sequences),  and  by  the  rough  agreement  between 
ascending  and  descending  patterns.  If  a  partial  match  is  obtained  (i.e.,  a 
close  match  between  subpattems),  it  will  constrain  further  transformations, 
restricting  them  to  the  non-matching  parts.  The  control  loop  will  influence 
the  process  by  selecting  the  patterns  to  be  expanded  and  the  sub-structmes 
participating  in  the  computation. 

A  close  match  between  corresponding  patterns  will  lead  to  the  establish¬ 
ment  of  a  complete  hnk  between  the  incoming  and  a  stored  representation. 
The  connections  participating  in  the  link  will  be  reinforced  to  facilitate  the 
future  processing  of  similar  patterns. 

PART  II 

Biological  Implications 

This  part  will  focus  on  the  biological  implications  of  the  sequence¬ 
seeking  scheme  and  its  implementation  in  the  coimter-stream  structure. 
The  discussion  is  divided  into  two  parts.  The  first  examines  the  implica¬ 
tions  of  the  model  for  the  general  connectivity  patterns  of  the  cortex,  both 
between  different  areas  and  between  layers  of  the  same  area.  The  second 
discusses  properties  of  the  different  types  of  connections  in  the  model.  The 
model  will  be  compared  with  known  connectivity  patterns  in  the  visual  cor¬ 
tex,  where  a  large  number  of  connections  have  been  studied  in  detail.  There 
is  evidence,  however,  that  other  cortical  areas,  including  somatosensory, 
motor,  and  auditory,  may  exhibit  generally  similar  connectivity  patterns 
(Felleman  &  Van  Essen,  1991). 

2.1  Connectivity  Patterns 

The  connections  between  cortical  areas,  as  well  as  between  layers  within 
an  area,  will  be  examined  and  interpreted  in  this  section  in  light  of  the 
sequence-seeking  model  discussed  in  Part  I.  As  we  shall  see,  the  proposed 
scheme  appears  to  be  in  general  agreement  with  the  existing  data,  and  it 
also  gives  rise  to  a  number  of  new  predictions.  The  discussion  involves  some 
fine  details  of  the  anatomy,  but  it  serves  to  illustrate  the  type  of  predictions. 
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on  both  general  and  specific  levels,  that  can  be  derived  firom  the  model. 
The  basic  connectivity  scheme 

A  skeleton  of  the  basic  connections  in  the  sequence-seeking  scheme  is 
shown  in  Fig.  3a.  It  is  composed  of  two  counter  streams,  an  ascending  one 
(A-*  B  -*  C  —*  D)  and  a  descending  one  (D  -*  C  —*  B  —*  A),  with  cross 
connections  (of  the  priming  type)  between  complementary  structures.  In  a 
biological  implementation,  each  of  the  patterns  ( A,  B,  etc.)  is  assumed  to 
be  a  sub-population  of  neurons,  acting  together,  possibly  with  some  degree 
of  synchrony.  A  connection  A  —*  B  means,  therefore,  that  a  certain  sub- 
poptilation  of  neurons  representing  pattern  A,  when  activated  together,  will 
activate  another  sub-population  B,  usually  in  a  different  layer  or  a  differen- 
t  cortical  area.  The  next  section  discusses  aspects  of  cortical  connectivity 
in  light  of  this  general  structure.  It  should  be  noted,  however,  that  the 
diagram  illustrates  schematically  the  activation  of  patterns  and  the  rela¬ 
tion  between  complementary  patterns.  The  actual  implementation  of  this 
scheme  in  terms  of  populations  of  neurons  is  likely  to  be  considerably  more 
complicated. 

In  terms  of  cortical  interconnections,  the  model  suggests  the  schematic 
connectivity  pattern  shown  in  Fig.  3b.  There  are  again  two  intercon¬ 
nected  coimter  streams.  The  ascending  one  goes  through  layer  4  to  a 
sub-population  of  the  superficial  layers,  denoted  in  the  figure  as  AS  (for 
Ascending  Superficial).  It  then  projects  to  layer  4  of  the  next  cortical 
area  (II  in  the  figure).  The  descending  stream  goes  through  a  different 
sub-population  of  the  superficial  layers  (DS,  for  Descending  Superficial)  to 
layer  6,  and  from  there  to  DS  of  a  preceding  area.  The  connections  can  also 
leap  over  one  step  in  the  stream,  e.g.,  AS  directly  to  AS  on  the  ascending 
stream,  and  DS  — »  DS  or  6  — »  6  on  the  descending  stream  (thin  lines  in 
Fig.  3b).  Layer  5  is  not  depicted  in  the  diagram  because,  according  to 
the  model,  it  is  involved  primarily  not  in  the  main  streams,  but  with  their 
control,  via  sub-cortical  areas  such  as  the  pulvinar. 

There  are  two  reasons  for  the  assumption  that  layer  5  (or  parts  of  it) 
may  be  involved  in  the  ‘control  loop’  mentioned  in  Part  I.  The  first  reason 
is  that  layer  5  (or  parts  of  it,  such  as  sublayer  5B  in  the  macaque’s  primary 
visual  cortex)  projects  in  an  orderly  topographic  manner  to  a  number  of 
subcortical  structures,  such  as  (in  the  visual  cortex)  the  pulvinar,  and  the 
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Figure  3a.  The  main  connections  along  and  between  streams. 
Full  arrows  denote  the  connections  along  the  stream,  unfilled 
etrrows  the  priming  connections  between  corresponding  patterns 
in  the  complementeury  streams. 
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Figure  3b.  The  basic  counter-stream  scheme,  showing  the  ^in 
connections  within  and  between  cortical  areas.  The  ascending 
path  goes  through  layer  4  and  the  ascending  superficial 
population  (AS)  to  the  next  area.  The  descending  path  goes 
from  the  descending  superficial  population  (DS)  to  layer  6 
and  to  the  lower  area.  The  connections  between  the  streams 
are  of  the  priming  type.  The  thin  arrows  show  pathways  that 
"leap  over”  one  step  in  the  stream. 


superior  colliculus.  The  pulvinar  is  reciprocally  connected  in  a  topographic 
manner  to  multiple  visual  areas.  This  pattern  of  connectivity  suggests  that 
some  of  the  subcortical  areas  may  be  involved  in  the  integration  and  control 
of  the  activity  in  different  visual  areas,  and  that  layer  5  is  a  part  of  this 
control  loop. 

The  second  piece  of  evidence  that  might  implicate  layer  5  with  some 
control  functions  is  the  finding  that  many  layer  5  pyramidal  neurons  show 
a  special  characteristic  firing  pattern  (Silva  et  al.  1991,  Kasper  et  al.  1991). 
These  cells  were  shown  to  have  a  tendency  to  produce  rhythmic  and  locally 
coordinated  firing  pattern.  Silva  et  al.  suggested  on  the  basis  of  their 
observations  that  “...neurons  in  layer  5  can  initiate  s3mchronized  rhythms 
and  project  them  on  neurons  in  all  layers”  (p.  434).  The  anatomical  and 
physiological  evidence  is  only  partial,  but  it  suggests  that  layer  5  cells  may 
play  a  control  and  coordination  role. 

The  scheme  diagrammed  in  Fig.  3b  implicates  layer  6  with  the  ascend¬ 
ing  stream  and  layer  5  primarily  with  a  separate  control  function.  This 
division  may  in  reality  be  less  clear-cut.  For  example,  parts  of  layer  5  may 
be  situated  on  the  descending  path  (Ferster  and  Lindstrom  1985),  and  parts 
of  layer  6  may  perhaps  participate  in  the  control  loop.  (For  example,  a  pop¬ 
ulation  of  layer  6  cells  projects  in  a  topographical  manner  to  the  claustrum, 
which  is  in  turn  reciprocally  connected  to  multiple  visual  areas.)  It  may  be 
more  appropriate,  therefore,  to  r^er  to  the  role  of  layer  6  in  the  diagram 
as  “DF,  for  “Descending  Infra”,  similar  to  the  DS  and  AS  populations  in 
the  superficial  layers.  The  diagram  is  also  schematic  and  over-simplified  in 
other  respects.  For  example,  there  may  be  additional  steps  within  an  area 
(as  in  the  parvo  stream  in  VI).  However,  the  main  goal  of  the  diagram 
is  not  to  accovmt  for  possible  variations,  but  to  emphasize  the  common 
xmderlying  structure  according  to  the  model. 

Connections  of  VI:  data  and  predictions 

Figure  4a,b  shows  a  more  detailed  version  of  this  diagram,  applied  to 
cortical  area  VI  (for  which  the  data  are  more  complete  than  for  other  visual 
areas),  and  its  cmmections  to  the  LGN  and  cortical  area  V2.  Fig.  4a  shows 
the  coimections  in  the  macaque  of  the  magno-cellular  stream,  and  4b  of 
the  parvo-cellular  stream,  on  the  way  from  the  LGN  (where  the  ascending 


24 


VI 


LGN 

magno 


LGN 


V2 


V2 


Figure  4a.  The  main  connections  according  to  the  model 
along  the  two  streams  from  the  LGN  (magno)  through  area 
VI  to  V2.  Thick  arrows:  established  connections.  Thin 
arrows:  connections  predicted  by  the  model. 


VI 


LGN 

parvo 


LGN 


V2 


V2 


Figure  4b.  The  main  connections  according  to  the  model 
along  the  two  streams  from  the  LGN  (parvo)  through  area 
VI  to  V2.  Thick  arrows:  esteUt>lished  connections.  Thin 
arrows:  connections  predicted  by  the  model. 


and  descending  streams  in  fact  converge)  to  area  V2  (e.g.,  Rockland  and 
Lund  1983,  Limd  1988a,  1988b,  Martin  1988).  The  division  into  the  two 
streams  may  be  somewhat  schematic;  there  is  evidence  (Limd  1988a)  for 
some  degree  of  Tniving  between  them. 

The  diagram  shows  the  main  connections;  there  is  evidence  for  addi¬ 
tional  secondary  ones  that  will  not  be  discussed  in  detail.  They  include: 
(i)  sparse  magnocellular  projections  to  the  lower  half  of  layer  6  (Blasdel  & 
Lund  1983),  (ii)  a  parvocellular  projection  to  4A  with  coUaterals  also  to 
3B  (Fitzpatrick  tt  al  1983,  Blasdel  &  Lund  1983),  (iii)  some  parvocelltilar 
input  to  the  upper  part  of  layer  6  (Hendrickson  et  al.  1978),  with  possible 
collateral  to  layer  1  (Blasdel  &  Lund  1983),  (iv)  projection  to  layers  2-3  of 
the  cytochrome  oxidase-rich  blobs,  (v)  some  possible  projection  &om  the 
upper  part  of  layer  6  and  lower  5  to  V2  (Kennedy  ii  Bullier  1985). 

The  diagram  includes  both  known  connections  and  connections  predict¬ 
ed  by  the  proposed  scheme  that  have  not  been  described  in  the  literature. 
The  predicted  coimections  are  discussed  in  the  next  three  subsections.  The 
first  relates  to  a  connection  within  the  descending  stream,  the  next  two  to 
the  reciprocal  connections  between  the  two  streams. 

The  Connections  from  the  Superficial  Layer  to  Layer  6 

The  model  pre  dicts  an  excitatory  connection  fiom  the  superficial  layers 
to  layer  6.  This  is  a  part  of  the  descending  stream  in  the  model,  firom 
population  DS  to  layer  6  in  Fig.  3b.  Such  a  connection  appears  physio¬ 
logically  possible,  since  the  apical  dendrites  of  layer  6  pyramidal  neurons 
projecting  back  to  the  LGN  extend  to  the  superficial  layers  (Martin  1988, 
Fitzpatrick  and  Einstein  1989)  where  they  can  make  connections  with  the 
DS  population. 

Interconnections  in  the  Superficial  Layer 

In  the  magnocellular  stream,  the  model  predicts  reciprocal  interconnec¬ 
tions  between  layer  4B  (playing  the  part  of  AS  in  the  model),  and  layers 
1-3,  the  recipients  of  descending  projections  firom  V2  (DS  in  the  model). 
The  projection  firom  4B  to  the  superficial  layers  is  well  established.  It  is 
also  known  (Lund  1988a)  that  4B  pyramidal  cells  send  apical  dendrites  to 
the  superficial  layers  where  the  coimection  may  take  place. 
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Lund  (1988a)  notes  a  ptizzle  regarding  the  connections  between  AS  and 
the  superficial  layers:  cells  in  4B  are  strongly  directionally  selective,  but 
the  superficial  layers  that  receive  input  from  them  are  markedly  less  di¬ 
rectionally  selective.  The  superficial  layers  show  strong  color  coding,  but 
cells  in  4B  do  not.  Such  discrepancies  are  in  fact  compatible  with  the 
counter-streams  scheme.  According  to  the  model,  these  two  popxilations 
are  situated  on  complementary  streams,  with  priming  coimections  between 
them.  It  is  therefore  not  expected  in  this  scheme  that  the  prevalent  prop¬ 
erties  of  one  of  these  populations  will  be  strongly  determined  by  the  other. 
The  superficial  layers  also  contain  other  sub-poptdations,  in  particular,  the 
AS  population  of  the  parvocellular  stream.  In  the  model,  the  color-coded 
non  directional  cells  in  the  superficial  layers  are  likely  to  be  a  part  of  this 
stream. 

There  is  another  specific  prediction  suggested  by  the  model  regarding 
these  connections.  As  can  be  seen  ^m  the  diagram,  the  expectation  from 
the  model  is  that  the  superficial  cells  connected  to  4B  will  also  be  the 
recipients  of  descending  projections  from  V2. 

InieTconnecUoru  Between  Layers  4  ^ 

The  counter  stream  model  in  Fig.  4a,  4b,  predicts  a  reciprocal  connec¬ 
tion  between  layer  4,  and  the  LGN-projecting  cells  in  layer  6. 

Layer  4C  in  the  macaque  is  known  to  receive  excitatory  connections 
from  layer  6  (Limd  1988a);  similar  connections  are  also  known  to  exist  in 
the  cat  primary  visual  cortex  (McGuire  et  al,  1984).  There  is  also  evidence 
for  the  existence  of  connection  in  the  opposite  directions,  as  reviewed  briefly 
below. 

In  the  tree  shrew,  connections  of  this  type  (from  4A/3B)  have  been 
reported,  (Lund  et  al.  1985).  In  the  magnocellular  stream  of  the  macaque, 
Lund  &  Boothe  (1975)  foimd  that  there  is  a  population  of  layer  6  pyramidal 
cells  with  arbors  of  their  apical  dendrites  in  4Ca  where  they  may  collect 
input  from  axons  of  layer  4Ca  cells,  and  also  layer  6  pyramidal  cells  with 
dendritic  arbors  in  4B,  the  next  station  in  Fig.  4a  along  the  ascending 
stream.  Th^  also  report  axonal  projections  from  layer  4B  to  layer  6.  This 
strongly  suggests  that  there  are  indeed  anatomical  routes  for  connecting 
two  stations  along  the  ascending  stream  (4Ca  and  4B)  to  layer  6. 
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In  the  parvocellular  system,  they  report  layer  6  pyramidal  cells  with 
dendritic  arbors  in  4A  and  3B,  sending  axons  back  to  these  layers,  with 
collaterals  in  AC0. 

This  manner  of  reciprocal  connections  between  layers  4  and  6  is  consis¬ 
tent  with  the  requirements  of  the  model.  The  main  role  assigned  to  these 
connections  by  the  model  is  to  provide  cross  connections,  presumably  of 
the  priming  type,  between  the  two  streams. 

It  also  interesting  to  note  in  this  regard  that  the  population  of  layer  6 
cells  projecting  back  to  the  LGN,  were  found  (in  the  cat)  to  be  the  same 
cells  that  are  also  connected  to  layer  4C,  by  axonal  collaterals  and  dendritic 
arbors  (Katz,  Burkhalter,  &  Dreyer  1984),  in  accordance  with  the  model 
in  Figs.  4a, b. 

General  connectivity  pattern:  data  and  predictions 

The  previous  section  focused  on  the  connectivity  of  the  primary  visual 
cortex.  This  section  examines  the  general  connectivity  pattern  within  and 
in  between  areas  in  light  of  the  sequence-seeking  scheme. 

Connections  Wiihin  an  Area 

If  the  hypothesis  regarding  the  coimter-streams  structure  is  broadly 
correct,  then  a  number  of  predictions  can  be  made  regarding  the  main 
connectivity  patterns  within  an  area.  The  predictions  listed  below  are 
for  the  most  direct  implementation  of  the  coimter-streazns  structure,  some 
modifications  and  exceptions  are  possible  in  some  cases. 

One  prediction  is  the  distinction  within  the  superficial  layers  between 
the  AS  and  DS  sub-populations.  The  AS  population  on  the  ascending 
stream  projects  to  a  higher  visual  area,  and  the  DS  poptilation  on  the 
descending  stream  is  the  target  of  back  projections  from  the  higher  area.  A 
number  of  different  AS  and  DS  popxilaticms  will  exist  in  parallel  within  the 
same  area,  e.g.,  projecting  to  (and  receiving  from)  distinct  targets.  This 
scheme  stands  in  contrast  to  an  alternative  view  (Sandell  &  Schiller  1982, 
Maunsell  &  Van  Essen  1983,  Rockland  &  Virga  1989)  in  which  the  back 
projections  contact  and  modulate  their  own  source. 

A  separation  between  the  two  popiilations  along  the  lines  suggested 
above  reflects  the  most  straightforward  implementation  of  the  scheme.  Such 
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a  separation  is  not  easy  to  establish  empirically,  but,  as  reviewed  below, 
at  least  in  some  cases  that  can  be  examined,  it  seems  to  be  honored  by 
the  system.  It  should  be  noted,  however,  that  some  exceptions  can  exist 
without  violating  the  constraints  of  the  model.  For  example,  priming  con¬ 
nections  can  connect  a  population  on  one  stream  with  a  population  on  the 
complementary  stream  in  a  different  visual  area.  As  a  result,  an  AS  pop¬ 
ulation,  for  instance,  can  be  both  the  source  of  ascending  connections,  as 
well  as  the  recipient  of  (priming)  connections  from  the  higher  area.  If  such 
connections  exist,  they  will  make  the  empirical  testing  of  the  separation 
more  difficult. 

The  separation  between  ascending  and  descending  populations  is  evi¬ 
dent  in  the  connections  involving  layer  4:  the  ascending  streams  terminate 
in  layer  4,  the  descending  streams  always  avoid  it.  In  the  superficial  layers 
the  situation  is  more  difficult  to  assess,  and  the  available  evidence  is  at 
present  restricted. 

An  example  involving  the  superficial  layers  is  the  magnocellular  projec¬ 
tion  from  VI  to  V2.  Rockland  and  Virga  (1989)  find  it  surprising  that  there 
are  no  terminations  of  back  projections  from  area  V2  to  layer  4B  of  VI  (the 
source  of  efferents  to  V2).  As  illustrated  in  Fig.  3b,  this  is  in  fact  to  be 
expected  in  the  current  model.  It  is  further  expected  that  even  in  those 
cases  where  the  same  layer  is  both  the  source  and  the  target  of  connections 
to  another  area  (such  as  the  connections  between  layer  4B  of  VI  and  MT), 
there  are  likely  to  be  two  distinct  sub-populations,  the  AS  and  DS,  within 
the  layer.  The  projection  from  layer  4B  of  VI  to  MT  was  studied  by  Zeki 
and  Shipp  (1988).  Retrograde  labding  revealed  a  patchy  origin,  suggesting 
that  only  a  sub-population  of  4B  neurons  participate  in  the  projection  to 
MT.  It  would  be  of  interest  to  know  whether  the  back  projection  from  MT 
is  directed  primarily  towards  the  complementary  sub-population. 

A  second  prediction  concerns  reciprocal  connections  within  an  area. 
That  is,  the  AS  and  DS  subpopulations  vdthin  the  same  area  are  expected  to 
be  connected  in  a  reciprocal  manner.  Similarly,  layer  6  (or  a  sub-population 
on  the  descending  path)  is  expected  to  be  connected  reciprocally  with  the 
ascending  path,  in  particular,  with  layer  4.  These  connections  are  expected 
to  be  of  the  priming  type  (Section  2.2). 

A  third  prediction  is  that  a  connection  is  expected  on  the  descending 
path  between  the  superficial  layers  (sub-population  DS)  and  layer  6  (or 


28 


the  DI  population).  (This  connection  may  become  less  important  when 
there  is  a  direct  connection  from  layer  6  of  the  higher  area  to  layer  6  of  the 
lower  area,  one  of  the  “leaping”  cotmections  in  Fig.  3b).  This  connection 
is  expected  to  be  of  the  “standard”  excitatory  type  (Section  2.2). 

ConntcHom  Between  Areas 

We  next  examine  the  general  patterns  of  connectivity  between  cortical 
areas.  It  will  be  convenient  in  examining  these  connections  to  follow  the 
classification  of  projections  between  areas  into  forward,  backward,  and  lat¬ 
eral  connections  (Rockland  ii  Pandya  1979,  Maunsell  Van  Essen  1983, 
Friedman  1983,  Van  Essen  1985,  Andersen  et  al.  1990,  Boussaud  et  al. 
1990,  Felleman  &  Van  Essen  1991).  It  has  been  observed  in  these  and 
other  studies  that  the  coimections  between  cortical  areas  (not  only  visual, 
but  also  somatosensory  and  motor)  can  be  classified  into  “forward,”  “back¬ 
ward,”  and  “lateral”  connections,  on  the  basis  of  the  laminar  distribution 
of  their  source  and  destination.  Roughly  speaking,  the  distinction  is  the 
following  (more  details  are  given  below).  Connections  of  the  forward  type 
terminate  in  layer  4,  and  (with  a  few  exceptions)  originate  in  the  superficial 
layers.  Backward  coimections  avoid  layer  4,  and  terminate  instead  in  the 
supra  and  infra  layers.  Lateral  connections  terminate  in  all  layers  (layer  4, 
as  well  as  supra  and  infra). 

When  the  connection  from  a  given  area  A  to  a  second  area  B  is  of 
the  forward  type,  the  connection  from  B  to  A  is  almost  invariably  of  the 
backward  type,  and  area  B  can  then  be  considered  ‘above’  A.  This  ordi¬ 
nal  relation  can  serve  to  suggest  a  hierarchy  of  cortical  areas,  which  has 
been  recently  reviewed  in  detail  by  Felleman  and  Van  Essen  (1991).  In  the 
sequence-seeking  scheme,  there  is  also  a  distinction  between  a  “forward” 
connection  (along  the  ascending  stream),  and  a  “backward”  connection 
(along  the  descending  one).  It  is  also  possible  in  the  scheme  for  these  con¬ 
nections  to  co-exist  (i.e.,  areas  A  and  B  can  be  connected  by  both  forward 
and  backward  connections),  which  will  give  rise  to  lateral  connections. 

Although  in  visual  processing  there  is  a  natural  general  hierarchy  of 
processing  from  the  incoming  signal  to  higher,  more  abstract  levels,  the 
sequence-seeking  model  by  itself  does  not  require  strict  hierarchy,  and  can 
exist  in  a  more  general  structure  of  redprocal  connections.  Violations  of 
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the  strict  hierarchy  are  possible,  and  are  more  likely  to  occur  at  the  higher, 
more  abstract  levels  of  processing.  In  the  discussion  below,  the  notion  of  a 
hierarchy  does  not  play  a  crucial  role,  but  the  classification  of  the  connection 
types  into  forward,  backward,  and  lateral,  serves  a  useful  role  in  organizing 
the  discussion.  As  we  shall  see,  the  somewhat  complex  set  of  possible 
projection  patterns  by  and  large  be  deduced  from  the  coimter-streams 
structure  diagrammed  in  Fig.  3b.  In  addition,  this  structure  may  help  to 
resolve  a  number  of  apparent  exceptions  to  some  general  connectivity  rules. 

The  Forward  Connection 

The  typical  forward  connection  between  cortical  areas  is  from  the  su¬ 
perficial  layers  of  one  area  to  layer  4  of  the  next  area  (Van  Essen  1985). 
This  pattern  of  connections  is  consistent  with  the  ascending  pattern  in  Fig. 
3b.  The  figure  also  shows  the  possibility  of  a  connection  that  leaps  over  one 
stage  in  the  stream,  from  AS  of  a  given  area  to  AS  of  another  (in  addition 
to  the  projection  to  layer  4).  Such  connections  have  indeed  been  observed 
(e.g.,  Rockland  1990;  the  parvocellular  projection  to  layer  4A  of  VI  may 
also  be  of  this  type). 

In  addition  to  this  main  pattern,  some  forward  connections  have  been 
foimd  to  originate  from  both  the  supra  and  infra  layers  (Andersen  ei  al. 
1990,  Felleman  &  Van  Essen  1991).  The  ascending  projection  from  the 
infra  layers  is  not  a  part  of  the  counter-streams  structure  of  Fig.  3b.  There 
are  several  possibilities  to  account  for  such  a  connection  within  the  counter- 
streams  pattern  of  connections.  For  example,  it  may  be  that  the  connection 
is  in  fact  a  lateral  rather  than  a  forward  one  (see  also  the  discussion  of  the 
lateral  connection  below).  A  lateral  connection  will  contain  an  ascending 
connection  as  well  as  a  descending  one,  that  will  supply  the  infra  -*  su¬ 
perficial  component.  The  result  will  be  a  bi-laminar  connection  (from  both 
supra  and  infra  layers)  of  one  area  to  layer  4  and  the  DS  layer  of  anoth¬ 
er.  This  may  be  the  case,  for  example,  in  the  projection  from  V3A  -* 
MT,  which  has  the  unusual  forward  connection,  and  where  the  opposite 
projection  MT  — »  V3A  is  suspected,  in  fact,  to  be  of  the  lateral  type. 

What  should  be  excluded,  according  to  the  model,  is  that  the  back- 
projecting  neurons  of  layer  6  will  also  be  the  origin  of  an  excitatory  ascend¬ 
ing  projection.  (They  could,  however,  be  the  source  of  priming  connections 
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to  the  ascending  stream.) 

The  Backward  Connection 

The  backward  connection  is  characterized  by  Felleman  &  Van  Essen 
(1991)  as  having  one  of  two  patterns,  which  they  denote  as  I  — »  M  or  B  — » 
M.  (I  denotes  infra,  B  denotes  bi-laminar,  and  M  denotes  termination  that 
avoids  layer  4.)  The  first  type  of  connection  is  from  the  infra  layers  (I)  to 
both  infra  and  supra  layers,  but  avoiding  4  (M).  The  second  type  is  from 
both  infra  and  supra  layers  (B),  again  to  M  (avoiding  4).  In  the  coimter- 
streams  pattern  of  connections  (Fig.  3b),  the  back  projection  is  composed 
of  the  connection  6  — »  DS,  and  the  “leaping”  connections  (DS  — ♦  DS,  6  — *  6, 
and  perhaps  also  DS  —*  6).  This  will  give  rise  to  both  of  the  patterns  listed 
by  Felleman  and  Van  Essen  (1991).  It  also  suggests  possible  additional  sub¬ 
cases,  for  example  a  connection  6  — ^  DS  only.  The  back  projection  from  V2 
to  VI  in  the  macaque  may  be  of  this  type.  Reconstructions  by  Rockland 
and  Virga  (1989)  show  projections  to  the  superficial  layers  (mainly  1  and 
2),  with  restricted  collaterals  in  layer  5  (but  not  in  6). 

The  Lateral  Connection 

Lateral  connections  in  the  hierarchy  detailed  by  Felleman  and  Van  Essen 
(1991)  are  projections  that  do  not  fit  the  forward  or  backward  types.  In 
their  terminology,  the  pattern  of  lateral  connections  is  B  — ♦  C;  the  origin 
is  bi-laminar,  both  supra  and  infra  layers  (B),  and  the  target  is  columnar, 
spanning  all  layers  (C).  The  pattern  of  connections  appears  complex  and 
sometimes  perplexing,  but  many  aspects  of  it  can  be  derived  almost  directly 
from  the  general  counter  streams-scheme.  In  the  counter-streams  model 
lateral  connections  are  in  fact  expected,  and,  in  terms  of  the  pattern  of 
Fig.  3b,  the  suggestion  is  that  the  lateral  connection  is  simply  the  “union” 
of  ascending  and  descending  connectioxis.  If  this  view  is  correct,  then  the 
Tnain  connections  participating  in  the  lateral  connection  can  be  inferred 
immediately  from  the  basic  scheme  (Fig.  3b).  They  include  the  direct 
connections: 

s  AS  4 
•  6-^DS 
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and  the  coxmections  that  leap  over  one  step: 

•  AS  — »  AS 

•  DS  -♦  DS 

•  6  — ♦  6 

It  can  be  seen  that  the  origin  of  the  projections  would  indeed  be  bi- 
laminar,  and  the  terminations  would  span  all  layers.  If  this  view  of  the 
lateral  connection  is  correct,  it  gives  rise  to  some  predictions,  and  offers  a 
possible  explanation  for  difficulties  noted  in  the  past,  and  discussed  below. 

The  Problem  of  Irregular  Terminations 

This  difficulty  arises  when  a  projection  from  one  area  A  to  another  area 
B  shows  a  mixed  pattern  of  terminations,  that  are  incompatible,  according 
to  the  hierarchical  view.  For  example,  when  some  of  the  terminations  are 
restricted  to  layer  4  of  area  B,  while  other  labelings  show  columnar  termi¬ 
nations.  This  mixed  pattern  is  denoted  F/C  by  Felleman  and  Van  Essen 
(F  for  termination  in  layer  four,  C  for  columnar  termination).  Termination 
in  layer  4  only  is  a  signature  for  connections  of  the  ascending  type,  but  a 
columnar  termination  signifies  a  connection  of  the  lateral  type  (Felleman 
&  Van  Essen  1991).  How,  then,  to  classify  the  A  —*  B  connection  in  this 
case? 

The  view  from  the  coimter-streams  structure  suggests  that  the  connec¬ 
tions  from  the  superficial  layers  of  A  to  area  B  in  a  lateral  coxmection  are 
composed  of  two  projections; 

•  ascending:  AS  — »  4 

•  descending:  DS  —*  DS,  6 

(In  addition,  there  is  a  descending  connection  6  -*  DS.) 

Anterograde  labeling  of  the  upper  layers  of  area  A  can  therefore  show 
a  mixed  pattern  of  terminations.  (In  anterograde  labeling,  dye  is  carried 
forward  along  cells  axons  and  across  the  synapses  to  the  post-synaptic  cells.) 
The  termination  pattern  may  be  either  in  4  alone  (from  the  AS  population 
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of  i4),  or  a  coltmmar  termination  (from  AS  and  DS).  This  may  be  the  F/C 
paradoxical  termination  noted  by  Felleman  and  Van  Essen.  It  can  also 
(by  labeling  the  DS  sub-poptilation  alone)  show  a  bi-laminar  pattern  of 
connections  in  both  the  supra  and  the  infra  granular  layers,  but  avoiding 
layer  4.  This  can  account  for  the  other  types  of  irregular  terminations. 

A  related  prediction  is  that  in  a  lateral  connection  between  areas  A 
and  B,  two  distinct  sub-populations  should  exist  in  the  superficial  layers  of 
B.  One  is  the  AS,  the  origin  of  ascending  connections  to  A,  the  second  is 
the  DS,  the  target  of  descending  connections  from  A.  The  two  should  be 
distinct  and  interconnected  (by  connections  of  the  priming  type). 

Origin^Terminaiion  Mismatch 

An  example  of  this  difficulty  (e.g.  from  area  V4  to  MT)  is  the  follow¬ 
ing.  In  the  connection  from  area  A  to  B,  the  termination  is  columnar  (all 
layers),  but  the  origin  is  sometimes  bi-laminar  (both  supra  and  infira),  and 
sometimes  only  superficial  (Felleman  &  Van  Essen  1991).  The  problem  is 
that  columnar  termination  signifies  a  lateral  type  of  connection,  which  is  in¬ 
consistent  (according  to  the  hierarchical  classification)  with  an  exclusively 
superficial  origin.  The  resolution  suggested  by  the  counter-streams  con¬ 
figuration  is  that  the  coimection  is  in  fact  of  the  lateral  type.  Retrograde 
labeling  of  the  superficial  layers  of  area  B  can  label  A  either  in  a  bi-laminar 
fashion  (from  the  DS  sub-population  of  B,  which  receives  input  from  both 
layers.  Fig.  3b),  or  just  the  superficial  layers  (from  the  AS  sub-population, 
which  receives  input  from  superficial  layers  only.  Fig.  3b). 

Finally,  it  is  worth  noting  that  the  coimter-streams  system  can  with¬ 
stand  the  elimination  of  some  connections  without  affecting  significantly  its 
sequence-seeking  function.  For  instance,  it  is  possible  to  have  connection 
of  the  lateral  type  in  the  projection  from  area  A  to  B,  and  at  the  same 
time  the  redprocal  connection  from  B  to  A  could  be  of  the  descending  (or 
ascending)  type  only.  (One  can  expect  a  higher  probability  of  removing  the 
ascending  link  in  the  higher  level  areas  and  the  descending  one  in  the  lower 
areas.) 

Why  six  layers? 

The  counter-streams  scheme  suggests  a  rough  functional  account  for  the 
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separation  of  the  cortex  into  five-to-six  main  layers  (five,  plus  the  somewhat 
special  layer  1).  Roughly  speaking,  within  this  model,  two  are  required  for 
the  ascending  stream,  two  for  the  descending  stream,  and  one  for  the  con¬ 
trol  loop.  (Layer  1  may  participate  in  the  control  loop  as  well  as  in  making 
some  of  the  intrinsic  connections  within  the  streams.)  The  layers  will  con¬ 
tain  sub-layers,  e.g.  different  sub-populations  of  AS,  projecting  to  different 
targets  on  the  ascending  stream,  and  similarly  for  the  DS  population  on 
the  descending  stream. 

The  division  is  not  precise,  because,  for  instance,  (i)  the  two  functional 
superficial  populations  (AS  and  DS  in  Fig.  3b)  may  not  be  always  seg¬ 
regated  in  a  consistent  manner,  (ii)  sometimes  (as  in  the  ascending  parvo 
stream  in  VI)  extra  sub-layers  may  be  required,  (iii)  layers  1  and  5  may  be 
involved  in  both  the  control  function  and  in  providing  other  connections. 
However,  variations  and  elaborations  are  naturally  likely  to  exist  in  such  a 
complex  system,  and  they  may  not  be  inconsistent  with  this  basic  account. 

2.2  Properties  of  the  Connections 

The  previous  section  examined  the  “wiring  diagram”  implied  by  the 
sequence-seeking  model.  In  this  section,  I  will  examine  properties  of  differ¬ 
ent  connections  implied  by  the  model. 

The  Role  of  inhibition 

The  model  makes  specific  predictions  concerning  the  excitatory  connec¬ 
tions  between  patterns.  However,  there  is  also  a  clear  role  for  inhibitory 
connections.  One  role  of  inhibition  is  within  the  basic  mappings,  or  trans¬ 
formations,  comprising  the  ascending  and  descending  streams.  Marr,  in 
his  models  of  the  neocortex  (Marr  1970)  and  archicortex  (Marr  1971),  sug¬ 
gested  two  functions  for  inhibitory  connections  in  the  mapping  between 
two  patterns,  represented  by  two  populations  of  neurons.  One  function  is 
to  ensure  that  the  number  of  co-active  output  units  is  approximately  con¬ 
stant.  This  is  obtained  by  affecting  their  threshold,  using  a  mechanism  of 
divisive  inhibition  (implemented  by  basket  cells).  The  second  function  is  to 
adjust  the  dynamic  range  of  the  output,  and  this  is  obtained  by  subtractive 
inhibition.  Such  inhibitory  connections  could  serve  similar  pvrposes  in  the 
coimter-streams  architecture. 
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In  addition  to  their  role  within  the  individual  mappings,  inhibitory  con¬ 
nections  are  likely  to  be  required  for  three  additional  roles.  The  first  is 
to  prevent  the  co-activation  of  too  many  patterns  (see  also  appendix).  A 
second  possible  function  is  inhibition  between  incompatible  sequences.  For 
example,  patterns  derived  in  the  ascending  stream  may  be  sufficient  to 
rule  out  certain  possible  interpretations,  and  it  would  be  useful  to  inhibit 
unlikely  sequences  from  expanding  further  in  the  descending  stream.  Fi¬ 
nally,  inhibition  may  play  a  role  in  the  control  loop  to  suppress  some  of  the 
sequences  from  expanding  further. 

Connection  types 

The  coimter-streams  structure  uses  excitatory  connections  for  a  number 
of  different  functions.  The  excitatory  connections  include  (i)  lateral  con¬ 
nections  among  neurons  belonging  to  the  same  patterns,  (ii)  connections 
between  successive  patterns  in  a  stream,  and  (iii)  priming  connections  be¬ 
tween  complementary  patterns  in  the  opposing  stream.  This  is  consistent 
with  the  observation  that  the  thalamic  inputs  to  the  primary  thalamic  re¬ 
cipient  cells  in  the  cortex  account  for  only  the  minority,  perhaps  25%,  of 
the  total  excitatory  inputs  to  a  single  neuron  (Lund  1988a). 

An  important  requirement  in  the  model  regarding  these  connection 
types  is  the  existence  of  priming  coxmections  between  the  complementary 
streams.  In  this  section,  I  disciiss  briefly  the  evidence  and  possible  mecha¬ 
nisms  for  these  connections. 

The  requirement  from  the  priming  connection  is  that  whenever  a  pattern 
A  in  the  ascending  stream  is  activated,  it  will  “prime”  the  corresponding 
pattern  A,  making  it  easier  to  activate  for  a  certain  time  period  follow¬ 
ing  the  stimulation.  Similarly  for  the  descending  stream,  A  will  activate 
its  counterpart  A.  Priming  interactions  of  this  predicted  type  have  not 
been  investigated  directly,  but  a  number  of  reported  findings  appear  to  be 
consistent  with  such  a  mechanism. 

Ferster  and  Lindstrom  (1985)  recorded  (both  extracellularly  and  intra- 
cellularly)  from  layer  4  in  the  cat’s  primary  visual  cortex  during  an  elec¬ 
trical  stimulation  of  the  LGN.  With  low-amplitude,  low-frequency  (1  Hz) 
stimulation,  the  response  was  mediated  only  by  the  geniculo-cortical  pro¬ 
jection  (from  the  LGN  to  layer  4).  At  a  higher  frequency  and  amplitude. 
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the  stimiilation  also  activated  layer  6  cells  by  antidromic  activation.  These 
cells,  via  their  collaterals,  add  excitatory  input  to  the  primary  layer  4  cells. 
These  layer  4  cells  now  receive  two  kinds  of  inputs  -  a  direct  one  from  the 
LGN,  and  a  second  one  from  the  collaterals  of  layer  6  cells.  Under  these 
conditions,  the  response  is  enhanced  significantly,  the  probability  of  firing 
increases,  and  most  cells  fire  multiple  spikes  in  response  to  each  stimulus. 
Under  the  opposite  conditions,  when  layer  6  is  inactivated,  the  main  ob¬ 
served  effect  is  the  reduction  in  excitability  of  layer  4  cells  (Grieve,  Miuphy 
&  Sillito,  1991).  These  findings  support  the  possibility  that  the  connection 
from  layer  6  to  layer  4  is  of  the  priming  type  in  the  model,  and  can  enhance 
the  response  of  layer  4  cells  in  the  ascending  stream. 

Some  indirect  support  for  the  possible  priming  of  layer  6  cells  by  the 
ascending  stream  comes  from  the  source-density  analysis  of  Mitzdorf  and 
Singer  (1978)  in  visual  areas  VI  and  V2  in  the  cat.  This  analysis  indicated 
that  the  activation  of  the  superficial  layers  by  an  ascending  stimulation  did 
not  cause  significant  activation  of  layer  6,  in  accordance  with  the  gener¬ 
al  separation  of  ascending  and  descending  streams.  The  main  observable 
effect  of  the  ascending  stream  on  layer  6  (in  addition  to  some  direct  L- 
GN  input)  appeared  to  be  a  long-lasting  subthreshold  excitatory  input. 
This  is  consistent  with  the  p<»sibility  of  a  priming  interaction  (see  also 
the  discussion  of  possible  mechanisms  below).  There  was  also  evidence  for 
long-lasting  subthreshold  excitatory  input  in  the  superficial  layers,  which, 
in  the  model,  is  attributable  to  the  priming  effect  of  the  ascending  stream 
on  the  complementary  DS  population. 

Interestingly,  in  a  similar  study  in  the  monkey  (Mitzdorf  &  Singer  1979), 
a  long  lasting  subthreshold  excitation  was  also  apparent  in  layer  4B  of  the 
primary  visual  cortex.  This  is  attributable  in  the  model  to  the  reciprocal 
coimections  between  layer  4B  in  area  MT.  According  to  the  model,  layer 
4B  contains  two  sub-populations  ,  one  on  the  ascending  stream  to  MT,  the 
other  on  the  complementary  descending  stream.  Ascending  stimulation  will 
then  cause  a  direct  excitation  of  the  ascending  sub-population,  followed  by 
a  long-lasting  excitation  of  the  complementary  descending  population. 

In  another  study  possibly  related  to  the  different  excitation  types,  Al- 
bowitz  and  Kuhnt  (1991),  using  voltage  sensitive  dyes,  observed  two  types 
of  excitation  patterns  in  the  visual  cortex:  a  fast  one,  followed  by  hyper¬ 
polarization,  and  a  slower,  gradual  one,  without  ensuing  hyperpolarization. 
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These  may  correspond  to  the  distinction  between  the  direct  and  priming 
excitatory  drives. 

Finally,  a  finding  regarding  priming  between  the  streams  comes  also 
from  studies  of  the  priming  of  LGN  neurons  by  projections  from  the  cor¬ 
tex.  Sur  and  his  collaborators  (Esguerra  &  Sur  1990)  have  examined  the 
response  in  LGN  slices  from  the  ferret  to  ascending  and  descending  stimu¬ 
lations  (from  the  optic  tract  and  optic  radiation,  respectively).  They  found 
that  when  a  descending  stimtdation  preceded  an  ascending  one,  it  enhanced 
the  response  to  the  ascending  stimulation  in  a  pronoimced,  non-linear  man¬ 
ner.  The  intracellular  responses  (EPSP)  were  larger  tmder  these  conditions 
than  the  algebraic  stun  of  the  individual  responses,  longer  in  time,  and  with 
higher  probability  of  driving  the  cells.  This  is,  again,  consistent  with  the 
priming  interaction  in  the  model  between  the  descending  and  ascending 
streams. 

Possible  mechanisms 

There  are  reasons  to  believe  that  the  priming  of  one  projection  by  an¬ 
other  is  a  plausible  neuronal  mechanism.  Evidence  for  two  possible  mecha¬ 
nisms  of  this  type  has  been  reported.  One  mechanism  has  been  investigated 
by  Miller,  Chapman,  and  Stryker  (1989).  In  this  study,  responses  of  cells 
in  the  cat’s  visual  cortex  to  visual  stimidaticm  were  profoimdly  suppressed 
by  the  blocking  of  NMDA  receptors  (by  using  APV).  A  possible  mechanis- 
m  proposed  by  Miller  ti  al.  by  which  NMDA  receptors  could  control  the 
responsiveness  of  cells  is  that  such  receptors,  when  activated  in  neocortex 
pyramidal  cells,  cause  a  slow,  long-lasting  EPSP  that  rises  to  a  peak  in  10- 
75  msecs.  As  a  consequence,  ’‘such  a  slow  EPSP  could  provide  a  base  upon 
which  continuing  subthreshold  input,  mediated  by  non-NMDA  receptors, 
would  become  suprathreshold”  (p.  5187). 

Another  mechanism  has  been  proposed  by  Koch  (1987)  and  has  received 
support  from  studies  by  Sur  and  his  collaborators  (e.g.,  Esguerra,  Kwon,  & 
Sur  1989)  and  by  Sherman  ti  al.  (1990)  in  the  LGN.  An  important  prop¬ 
erty  of  NMDA  receptors  is  that  they  can  increase  the  cell’s  response  in  a 
nonlinear  fashion,  as  a  fimction  of  the  depolarization  in  the  post-synaptic 
cell.  The  proposal  (Koch  1987)  is  therefore  that  the  descending  stimula¬ 
tion  of  the  LGN  causes  long-lasting  subthreshold  depolarization.  In  this 
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proposal,  the  ascending  stimulation  involves  receptors  of  the  NMDA  type. 
If  they  are  activated  while  the  units  are  still  in  a  depolarized  state,  the  re¬ 
sponse  will  be  enhanced  significantly.  The  long-lasting  depolarization  could 
be  contributed  by  post-synaptic  responses  with  slow  time  course,  similar  to 
the  persistent  Na^  channel,  or  the  It  calcium  channel  (McCormick  1990). 
They  can  also  be  contributed  by  the  activation  of  distal  parts  of  the  dendrit¬ 
ic  tree.  Simulations  of  pyramidal  cells  (Stratford  et  al.  1989)  have  shown 
that  such  stimulation  can  have  a  significant  temporal  extent.  It  is  also  of 
interest  to  note  in  this  regard  that  McGuire  et  al.  (1984)  have  found  that 
the  connections  from  layer  6  to  layer  4  in  the  cat  (assumed  in  the  model  to 
be  of  the  priming  type)  make  connections  to  distal,  slender  branches  of  the 
dendritic  tree. 

The  above  two  mechanisms  could  serve,  individually  or  in  combination, 
to  implement  connections  of  the  priming  t}rpe.  Other  mechanisms,  not 
considered  here,  might  participate  as  well.  Although  the  details  are  not 
known,  it  appears  that  synaptic  mechanisms  for  priming  connections  are 
physiologically  plausible. 

Modifiable  connections 

Several  of  the  excitatory  coimections  described  above  may  be  modifiable 
by  experience.  The  learning  and  reinforcement  of  sequences  discussed  in 
Part  I  suggests,  in  particular,  that  the  coimections  between  successive  pat¬ 
terns  in  a  stream  should  be  reinforced  following  the  successful  establishment 
of  a  complete  sequence. 

Two  predictions  follow.  First,  the  synaptic  connections  between  succes¬ 
sive  stages  in  each  stream,  such  as  AS  — » 4,  or  6  — »  DS,  should  be  modifiable. 
Second,  the  synaptic  reinforcement  should  be  particularly  effective  if  the 
stimulation  along  the  stream  is  preceded  by  a  stimulation  of  the  comple¬ 
mentary  stream.  Long-term  reinforcement  effects  have  been  reported  in 
the  visual  cortex  (Teyler  et  al.  1990),  but  they  are  sometimes  difficult 
to  elicit  (Artola  &  Singer  1987).  It  would  be  of  interest  to  test  whether 
some  of  these  effects  can  be  facilitated  by  an  appropriate  stimulation  of  the 
complementary  stream. 

Finally,  some  of  the  reported  LTP  effects  in  the  neocortex,  including 
those  that  persist  for  only  intermediate  time  intervals  (minutes  to  a  few 
hours)  rather  than  very  long  ones,  could  be  useful  in  implementing  context 
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effects  of  the  type  discussed  in  Part  I. 

The  feedback  projections 

One  of  the  most  characteristic  properties  of  the  visual  (as  well  as  non¬ 
visual,  Felleman  &  Van  Essen  1991)  cortex  is  the  reciprocity  of  the  con¬ 
nectivity  pattern.  As  a  general  rule,  whenever  area  A  sends  a  forward 
projection  to  area  B,  there  is  also  a  back-projection  from  B  to  A.  In  the 
sequence-seeking  scheme,  this  rule  follows  from  the  counter-streams  archi¬ 
tecture  (Fig.  3).  The  general  rule  in  this  scheme  is  that  every  forward 
connection  A  — »  B  is  accompanied  by  a  reciprocal  connection  B  —*  A. 

In  the  case  of  the  projection  from  the  LGN  to  the  primary  visual  cortex, 
there  is  a  possible  reason  why  the  back-projection  may  in  fact  be  consider¬ 
ably  more  extensive  than  the  forward  projection.  The  reason  is  related  to 
Mart’s  theory  of  the  cortex  (Mart  1970,  1971).  In  his  model.  Mart  found 
that  it  is  important  to  ensure  that  the  input  to  the  cortex  will  be  sparse; 
that  is,  that  the  number  of  active  fibers  in  the  population  encoding  of  a 
pattern  will  be  kept  relativdy  small.  (A  similar  conclusion  was  reached  by 
Willshaw  ti  ai,  1969.)  This  consideration  implies  that  at  the  input  stage 
from  the  LGN  to  the  cortex  it  is  desirable  to  re-encode  the  input  in  a  spars¬ 
er  manner  (this  may  be  one  of  the  functions  of  the  VI  granular  layer).  The 
result  will  be  a  mismatch  between  the  LGN  and  VI:  patterns  in  VI  will 
be  represented  by  a  larger  number  of  units,  and  the  number  of  descending 
connections  will  be  correspondingly  larger. 

Effects  of  the  feedback  projection 

According  to  the  sequence-seeking  scheme,  the  physiological  effects  of 
the  descending  projections  are  expected  to  be  of  two  distinct  types.  The 
first  effect  is  a  priming,  or  modulatory  effect.  After  a  descending  stream  has 
been  activated,  it  leaves  the  corresponding  ascending  stream  in  a  primed 
state,  making  it  more  responsive  to  visual  stimulation.  The  second  effect 
is  direct  activation:  a  pattern  on  the  descending  path  in  a  lower  area  can 
be  directly  activated  by  a  higher-level  pattern  in  the  stream. 

Both  of  these  effects  of  back-projections  have  in  fact  been  observed  in 
physiological  studies.  Modulatory  ^ects  have  been  described  by  Nault  et 
al.  (1990,  also  Sandell  &  Schiller  1982)  who  studied  the  effects  on  area  17 
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of  reversibly  inactivating  area  18  in  the  cat.  The  main  effect  found  was  a 
reduced  responsiveness  of  area  17  units  to  visual  stimulation  when  area  18 
was  inactivated.  Similar  modulatory  effects  have  been  found  in  the  LGN, 
induced  by  cortical  back-projections  (e.g.,  Geisert  et  al.  1981,  Marrocco 
et  al.  1982).  The  reported  effects  were  relatively  subtle,  and  were  both 
excitatory  and  inhibitory. 

A  prediction  of  the  model  regarding  the  modulatory  effects  is  that  facil¬ 
itation  by  the  back  projections  will  not  require  strict  temporal  coincidence: 
a  descending  activation  should  be  able  to  facilitate  an  ascending  one  even 
if  it  precedes  it  in  time.  Another  prediction  of  the  model  is  that  similar 
modulatory  effects  are  also  likely  to  be  induced  by  ascending  signals  on 
descending  ones. 

The  priming  effects  are  expected  in  the  model  to  be  subtle,  and  they 
could  lead  to  both  excitatory  and  inhibitory  effects.  An  activation  of  a 
descending  sequence  will  prime  the  complementary  patterns.  At  the  same 
time,  due  to  competition  between  patterns  and  inhibition  among  them  as 
discussed  above,  the  effect  on  other  imits  could  be  inhibitory.  At  the  level 
of  the  LGN,  the  effects  are  expected  to  be  mainly,  (perhaps  exclusively),  of 
the  modulatory  type. 

A  direct  excitatory  effect  of  the  descending  projections  was  fotmd  by 
Miguard  and  Malpeli  (1991).  They  have  blocked  the  direct  ascending  in¬ 
put  from  the  LGN  to  VI  in  the  cat,  and  found  that  layers  in  the  upper 
layers  of  VI  could  still  be  strongly  activated  via  the  back-projections  from 
V2.  They  conclude  that  their  findings  do  not  support  the  modulatory  view 
of  the  feedback  circuits,  since  the  back-projections  can  in  fact  drive  VI 
units  vigorously.  FVom  the  point  of  view  of  the  sequence-seeking  model, 
both  of  these  interactions  are  in  fact  expected.  Furthermore,  it  is  expected 
that  they  will  be  segregated  into  two  distinct  sub-populations,  one  on  the 
ascending,  the  other  on  the  descending  stream.  Referring  back  to  Fig.  2, 
pattern  B  on  the  descending  stream  can  be  directly  driven  from  higher  pat¬ 
terns  along  the  descending  stream.  In  contrast,  pattern  B  on  the  ascending 
stream  is  expected  to  show  priming,  modulatory  effects  of  the  descending 
stream. 

Due  to  the  direct  connections  on  the  descending  stream,  tmder  the  ap¬ 
propriate  conditions  one  may  expect  to  see  complex  patterns  of  activity 
induced  by  the  descending  stream  even  in  lower-level  visual  areas.  These 
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patterns  will  change  with  time,  as  sequences  that  have  been  explored  decay 
and  are  replaced  by  new  ones.  Such  patterns  may  be  difficult  to  observe 
with  single-cell  recordings,  but  they  may  be  observable  with  suitable  optical 
recording  techniques  with  voltage-sensitive  dyes. 

Conclusion 

In  the  sequence-seeking  model,  the  basic  operation  of  the  neocortex  is  to 
seek  sequences  of  transformations,  or  mappings,  linking  source  and  target 
representations.  This  operation  is  implemented  in  a  coimter-streams  struc¬ 
ture,  in  which  multiple  ‘bottom  up’  and  ‘top  down’  sequences  are  explored 
in  parallel  along  separate  ascending  and  descending  pathways.  The  indi¬ 
vidual  mappings  and  transformations  will  differ  in  different  cortical  areas, 
but  the  basic  flow  of  the  process,  the  parallel  bi-directional  exploration  of 
a  large  number  of  sequences,  is  assumed  to  be  common  to  different  parts 
of  the  cortex. 

Many  elaborations  and  local  variations  may  exist,  but  the  basic  scheme 
is  uniform  and  in  principle  straightforward.  Key  properties  that  are  ex¬ 
pected  to  contribute  to  the  scheme’s  power  include:  the  simultaneous  ex¬ 
ploration  of  a  large  number  of  alternative  sequences,  the  relative  simple, 
uniform,  and  extensible  structure,  the  flexible  use  of  ‘bottom  up’  and  ‘top 
down’  sequences  that  can  meet  at  any  level,  the  broad  and  indirect  context 
effects,  and  the  learning  of  complete  sequences  by  a  simple  local  reinforce¬ 
ment  rule. 
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APPENDIX 

Multiple  Co’Exuiing  Patterns 

In  the  sequence-seeking  scheme  it  is  desirable  to  explore  in  parallel  a 
large  number  of  different  sequences.  In  an  actual  implementation,  there 
will  be  constraints  on  the  maximal  possible  number  of  co-active  patterns. 
In  particular,  if  patterns  are  represented  by  the  activity  of  different  sub- 
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populations  of  neurons,  the  co-activation  of  too  many  patterns  will  lead  to 
interference  between  patterns  and  possible  confusion. 

In  this  appendix,  an  approximate  boimd  on  the  number  of  co-activated 
patterns  is  derived  in  the  context  of  a  simple  associative  memory  model,  of 
the  type  described  by  Willshaw,  Baneman,  and  Longuet-Higgins  (1969). 

In  this  model,  the  memory  condsts  of  n  input  and  n  output  neiurons, 
with  complete  interconnections.  Pairs  of  patterns  (a,  —  6,)  can  be  written 
into  this  memory.  This  means  that  if  any  of  the  patterns  a,-  is  presented 
as  an  input  to  the  system,  it  will  reproduce  the  corresponding  6,-  (even  if 
a,  is  partially  corrupted).  Willshaw  et  al.  have  shown  that  imder  certain 
assumptions,  their  model  can  store  about  /n2  x  n  difEerent  pattern  pairs. 

For  the  co-activation  of  pattens,  the  relevant  question  is  what  happens 
if  a  number  of  different  patterns  (a^,  a^, . . . ,  a^)  are  presented  to  the  system 
simultaneously.  We  would  like  to  get  at  the  system’s  output  all  of  the 
patterns  (&i,  63, . . . ,  6jk)  but  no  other  patterns.  The  question  is,  essentially, 
how  large  can  k  be  without  leading  to  possible  confusion? 

The  patterns  in  this  model  are  composed  of  I’s  (“active”  \mits)  and  O’s 
(“inactive”  ones).  Assume  that  k  patterns,  each  composed  of  m  I’s  (and 
n  —  m  O’s)  are  presented  simultaneously.  The  total  number  of  I’s  in  the 
input  can  be  estimated  as  follows.  The  probability  of  a  given  input  line  not 
being  activated  by  a  given  pattern  is  1  —  The  probability  of  not  being 
activated  k  independent  inputs  is  (for  m  <<  n): 

(1  _  'H)  w  c-®*  (^1) 

n 

and  the  number  M  of  active  input  units  is  therefore 

M  =  n(l  —  c~^*).  (i42) 

How  large  can  M  be  without  caxising  possible  problems?  We  would  like 
the  probability  of  an  output  pattern  b  bdng  activated  by  mistake  by  the 
input  to  be  small.  A  pattern  b  will  be  activated  by  chance  if  all  (or  almost 
all)  of  its  input  will  happen  to  be  activated.  The  probability  that  this  will 
happen  for  a  given  pattern  b  is  (JIf/n)’”.  To  keep  the  average  number  of 
such  errors  small,  we  will  require;  n  x  (A//n)”*  <  1  (since  there  are  close  to 
n  patterns  stored).  For  optimal  use  of  the  associative  memory,  Willshaw 
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tt  al.  have  determined  that  m  should  be  about  log^n.  This  leads  to  the 
requirement: 


M  <  0.5n 

M3). 

Combining  this  with  (A2): 

M  = 

n(l  —  )  =  0.5n 

M4) 

therefore: 

,  /n2  X  n 

(AS) 

k  = - . 

m 

Or,  assuming  that  m  =  log^n-. 

^  /n2  X  n 

(A6) 

log^n  ‘ 

In  this  model,  then,  the  maximal  number  of  patterns  that  can  be  stored 
is  P  =  ln2  X  n,  and  the  maximal  number  of  co-active  patterns  should  not 
exceed 


p 

Tog^' 


As  expected,  the  fact  that  the  patterns  are  not  represented  indepen¬ 
dently  places  constraints  on  the  number  of  patterns  that  can  be  activated 
simultaneously.  At  the  same  time  (at  least  within  this  associative  memory 
model),  the  nvunber  of  co-active  patterns  can  still  be  large. 

An  additional  constraint  worth  noting  is  where  the  mapping  from  one 
set  of  patterns  to  the  next  may  not  be  invertible.  For  example,  suppose  that 
patterns  are  being  classified  into  **s3rmmetric”  vs.  ‘*non-symmetric.*’  Given 
a  pattern,  the  ‘Yorward”  mi^ping  (i.e.,  dasdfying  it  as  symmetric  /  non- 
symmetric)  can  be  computed.  In  the  opposite  direction,  however,  knowing 
that  a  pattern  is  symmetric  is  insuffident  for  reconstructing  the  pattern. 
In  the  sequence-seeking  scheme,  it  was  assumed  that  if  a  transformation 
A  —*  B  can  take  place  in  the  ascending  stream,  then  the  transformation 
B  —*  A  can  take  place  in  the  descending  stream.  In  the  example  above,  the 
situation  is  asymmetric:  A-*  B  ia  well-defined,  while  the  inverse,  B  —*  A, 
is  not.  The  main  implication  of  this  constraint  is  to  reqtiire  certain  temporal 
ordering  on  the  expandon  of  sequences,  e.g.,  that  A-*  B  on  the  ascending 
stream  will  have  to  ocoir  before  B  Aon  the  descending  stream. 
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